Development tools

• Getting the code
• Developing with JMRI Code
• Git FAQs
• Building with:
  • Ant
  • NetBeans
  • Eclipse
  • IntelliJ IDEA

Code Structure

• Introduction
• Names and Naming
• Application Structure
• Code Structure Patterns
• Swing Structure
• External Connection Structure
• Application Preferences
• Profile Files
• Threading
• Use of XML
  • XML Schema
  • XSLT Formatting
• Web Site
• Javadocs
  • Name Index
  • Packages
  • Classes

Techniques and Standards

• Recommended Practices
• Continuous Integration
• Internationalization
• Logging & Error Handling
• Unit testing with JUnit
• Documentation using Javadoc and UML
• Run the SpotBugs static analysis tool
• Providing help using JavaHelp
• Portable Filename Method
• XML Persistance

Help and Web Site

• Providing Help using JavaHelp
• The JMRI Web Site
• Updating JMRI Web and Help
• Creating Help for New Hardware
• Program Documentation Using Javadoc

How To

• Extend JMRI
• Add a New System
• Add a New Type

Functional Info

• Turnout Feedback
• Network Connections
• How the startup scripts work

Background Info

• Technology Roadmap
• Survey of JVM Capabilities
• Usage Poll, Spring 2007

JMRI Help:
Contents/ Index
Glossary/ FAQ

Donate to JMRI.org

JMRI Code: Unit testing with JUnit

• Introduction
• Running the Tests
• Continuous Integration Test Execution
• Error Reporting
• Code Coverage Reports
• Writing Tests
• • Writing Additional Tests for an Existing Class
  • Writing Tests for a New Class
  • Writing Tests for a New Package
• Key Metaphors
• • AssertJ tools
  • Handling Log4J Output From Tests
  • Resetting the InstanceManager and other Managers
  • Working with Listeners
  • Working with Threads
  • Testing I/O
  • Temporary File Creation in Tests
  • JUnit Rules
  • • Timeout rule
    • RetryRule
  • Controlling JUnit Tests
• Testing Swing Code
• • Using AssertJ with Swing
  • Using Jemmy
• Testing Script Code
• • Testing Jython sample scripts
• Issues

For more information on JUnit, see the JUnit home page. We now use JUnit version 5 (JUnit5), although a lot of JMRI code originally had tests written in the previous versions, JUnit3 and Junit4.

A very interesting example of test-based development is available from Robert Martin's book.

All of the JMRI classes have JUnit tests available; we have decided that our Continuous Integration system will insist on that. It's good to add JUnit tests as you make changes to classes (they test your new functionality to make sure that it is working, and keeps working as other people change it later), when you have to figure out what somebody's code does (the test documents exactly what should happen!), and when you track down a bug (make sure it doesn't come back).

Running the Tests

   ant alltest

   ant tests
   ./runtest.csh jmri.jmrit.powerpanel.PowerPanelTest

There is also a PowerShell (Core) script available to help running tests with maven,

./java/runtests.ps1

• Find all test cases in a package, including sub packages
• Generate code coverage reports locally
• Optionally: Target a single test class inside the specified package
• Optionally: Clean, and recompile, all tests and objects under test

powershell -File .\java\runtests.ps1 -?

Get-Help .\java\runtests.ps1 -detailed

Optional Checks

jmri.skipschematests

If true, JUnit tests will skip checking the schema of all the test XML files.

jmri.skipjythontests

If true, JUnit tests will skip running the jython/tests scripts.

jmri.migrationtests

When set true, run some extra tests; usually used during code migration, where not everything is right yet but you want to be able to include tests for individual running.

jmri.util.JUnitUtil.checkSetUpTearDownSequence

If true, check for whether JUnitUtil.setUp() and JUnitUtil.teardown() follow each other int the proper sequence. Print a message if not. (This slows execution a bit due to the time needed to keep history for the message)

jmri.util.JUnitUtil.checkSequenceDumpsStack

If true, makes jmri.util.JUnitUtil.checkSetUpTearDownSequence more verbose by also including the current stack trace along with the traces of the most recent setUp and tearDown calls.

jmri.util.JUnitUtil.checkRemnantThreads

If true, checks for any threads that have not yet been terminated during the test tearDown processing. If found, the context is logged as a warning.

jmri.util.JUnitUtil.checkTestDuration

If true, issues a warning if a test takes too long. The default limit is 5000 msec, but you can change it defining the jmri.util.JUnitUtil.checkTestDurationMax environment variable.

Controlling Test Operation

jmri.demo

When set to "true", this tells certain tests to leave windows open so that they can be used as demos or manual tests. The default is "false", clean up at the end of the test.

jmri.jmrit.audio.DefaultAudioManager.implementation

When set, use the specified class for the AudioFactory implementation. Setting it to jmri.jmrit.audio.NullAudioFactory will recreate the audio environment on the CI machines, which have no audio hardware available.

jmri.log4jconfigfilename

Override the default "tests.lcf" logging control file name.

jmri.shutdownmanager

When set, use the specified class as the default ShutDownManager. The specified class must implement the jmri.ShutDownManager interface and have a public default constructor. This is used in our standard testing infrastructure(s) to ensure that a mocked test-compatible ShutDownManager is available.

jmri.skipTestsRequiringSeparateRunning

When set to "true", this skips running certain tests that should be run separately (not in the long series of tests we use for CI running) because they behave intermittently when run as part of the main test sequence. The scripts/run_flagged_tests_separately file runs these test classes separately.

jmri.util.JUnitUtil.printSetUpTearDownNames

If true, JUnit tests will print to stderr each JUnitUtil.setUp() call and when checkSetUpTearDownSequence is true will also print to stderr each JUnitUtil.teardown() call. This can be useful if i.e. the CI tests are hanging, and you can't figure out which test class is the problem.

jmri.util.junit.PrintingTestListener.verbose

Prints detailed lists of each test completed, failed and ignored to stdout. It's a lot of output, but can help track down when an obscure message was emitted or exception happened. It also can be used to see the order in which tests are being run.

jmri.util.junit.PrintingTestListener.quiet

Reduces the information printed during a normmal JUnit run to dots (.) for passed tests, F for failed ones, A for aborted ones (assumptions not met) and I for @Ignored ones. This is followed by a one-line summary, then more detail on failures.

A Note on Internationalization (I18N)

To avoid false failures, the Ant and Maven build control files set the locale to en_US before running tests. This covers continuous integration running, and running locally using e.g. "ant headlesstest" or "ant alltest".

The ./runtest.csh mechanism does not automatically set the locale. To do that, the easiest approach is to set the JMRI_OPTIONS environment variable via one of:

   setenv JMRI_OPTIONS "-Duser.language=en -Duser.region=US"
   export JMRI_OPTIONS="-Duser.language=en -Duser.region=US"

Continuous Integration Test Execution

Error Reporting

You may visit the web site to subscribe to the jmri-builds e-mail list to get the bad news as quickly as possible, or monitor to view the archives of the e-mail list and see past logs. Or you can monitor the "dashboard" at the continuous integration web site.

(When the build succeeds, nothing is mailed, to cut down on traffic)

Code Coverage Reports

• A plot of coverage as a whole. Click on the graph to see a
• summary by Java package. Click on a package to see a
• summary by file (e.g. class). Click on a class to see a
• summary by method. Click on a method to see
• how each part of the code was covered (may require scrolling down).

Writing Tests

Writing Additional Tests for an Existing Class

The JUnit conventions require that the test be preceeded by the "@Test" annotation:

    @Test
    public void testSomething() {
        ...
    }

In general, test methods should be small, testing just one piece of the classes operation. That's why they're called "unit" tests.

Writing Tests for a New Class

To write a test for a new class, you need to create a file that shadows your new class. For our example, consider creating a test for a new class that appears in "src/jmri/jmrix/foo/Foo.java". The new test would be created in a file named "test/jmri/jmrix/foo/FooTest.java".

Assuming that the Foo class has a default constructor named foo(), Then the following would be minimal contents for the test/jmri/jmrix/foo/FooTest.java file:

    package jmri.jmrix.foo;

    import jmri.util.JUnitUtil;
    import org.junit.jupiter.api.BeforeEach;
    import org.junit.jupiter.api.AfterEach;
    import org.junit.jupiter.api.Test;

    import static org.assertj.core.api.Assertions.assertThat;

    /*
     * Tests for the Foo Class
     * @author  Your Name   Copyright (C) 2016
     */
    public class FooTest {

        @Test
        public void testCtor() {
            assertThat
            Assert.AssertNotNull("Foo Constructor Return",new foo());
        }

        @BeforeEach
        public void setUp() {
            JUnitUtil.setUp();
        }

        @AfterEach
        public void tearDown() {
            JUnitUtil.tearDown();
        }
    }

Note that you should be invoking jmri.util.JUnitUtil.setUp() and jmri.util.JUnitUtil.tearDown() as above.

In addition, the tearDown() method should set all member variable references to null. This is because JUnit keeps the test class objects around until all the tests are complete, so any memory you allocate in one class can't be garbage collected until all the tests are done. Setting the references to null allows the objects to be collected. (If the allocated objects have a dispose() method or similar, you should call that too). You should not reset the InstanceManager or other managers in the tearDown method; any necessary manager resets will be done automatically, and duplicating those wastes test time.

You may also choose to copy an existing test file and make modifications to suite the needs of your new class.

Writing Tests for a New Package

Key Test Metaphors

AssertJ Tools

• Checking of file contents
• Checking that the correct exception is thrown:

        Throwable thrown = catchThrowable(() -> { testObject.throwAnException() });
  	  assertThat(thrown).isInstanceOf(NullPointerException.class)
  	                  .hasNoCause()
        }
        

Handling Log4J Output FromTests

Log4j usage in the test classes themselves has two aspects:

1. It's perfectly OK to use log.debug(...) statements to make it easy to debug problems in test statements. log.info(...) can be used sparingly to indicate normal progress, because it's normally turned off when running the tests.
2. In general, log.warn or log.error should only be used when the test then goes on to trigger a JUnit assertion or exception, because the fact that an error is being logged does not show up directly in the JUnit summary of results.

On the other hand, you might want to deliberately provoke errors in the code being tested to make sure that the conditions are being handled properly. This will often produce log.error(...) or log.warn(...) messages, which must be intercepted and checked.

To allow this, JMRI runs it's using tests with a special log4j appender, which stores messages so that the JUnit tests can look at them before they are forwarded to the log. There are two aspects to making this work:

1. All the test classes should include common code in their setUp() and tearDown() code to ensure that log4j is properly initiated, and that the custom appender is told when a test is beginning and ending.

       @BeforeEach
       public void setUp() throws Exception {
           JUnitUtil.setUp();
       }
       @AfterEach
       public void tearDown() throws Exception {
           JUnitUtil.tearDown();
       }
   

2. When a test is deliberately invoking a message, it should then use JUnitAppender class methods to check that the message was created. For example, if the class under test is expected to do
   

       log.warn("Provoked message");
   

   the invoking test case should follow the under-test calls that provoke that with the line:
   

       jmri.util.JUnitAppender.assertWarnMessage("Provoked message");
   

   It will be a JUnit error if a log.warn(...) or log.error(...) message is produced that isn't matched to a JUnitAppender.assertWarnMessage(...) call.

3. The Log4JUtil.warnOnce(..) requires some special handling in tests. We want each test to be independent, so we reset the "want only once" logic early in the JUnitUtil.setUp() that's routinely invoked @BeforeEach the tests. This means that the first invocation, and only the first invocation, for each message will be logged.
4. We want to make it easy to add a Log4JUtil.deprecationWarning call when a method is deprecated. This will log a message the first time it's invoked. We want to warn that deprecated code is being invoked during normal operation, so this is normally becomes a Log4JUtil.warnOnce(..) call. When you see those warnings messages, you should remove them by completing the migration away from the deprecated method.

   The one exception is during unit and CI testing of the actual deprecated method. We want to keep those tests around until the deprecated method is finally removed. That ensures it keeps working until it's deliberately removed, and not inadvertently broken in the meantime. In this case, you should turn off the Log4JUtil.deprecationWarning in just that test method using Log4JUtil.setDeprecatedLogging(false) before invoking the deprecated method. (You can also do an JUnitAppender.assertWarn for all the messages emitted, but it's easier to just turn them off.)

Note: Our CI test executables are configured to fail if any FATAL or ERROR messages are emitted instead of being handled. This means that although you can run your tests successfully on your own computer if they're emitting ERROR messages, but you won't be able to merge your code into the common repository until those are handled. It's currently OK to emit WARN-level messages during CI testing, but that will also be restricted (cause the test to fail) during tne 4.15.* development series, so please suppress or handle those messages too.

Resetting the InstanceManager

Depending on what managers your code needs, your setUp() implementation could start with:

    JUnitUtil.setUp();
    JUnitUtil.resetInstanceManager();
    JUnitUtil.resetProfileManager();
    JUnitUtil.initConfigureManager();

    JUnitUtil.initDebugCommandStation();

    JUnitUtil.initInternalTurnoutManager();
    JUnitUtil.initInternalLightManager();
    JUnitUtil.initInternalSensorManager();
    JUnitUtil.initReporterManager();

Your tearDown() should end with:

    JUnitUtil.tearDown();

Working with a ShutDownManager

• If you leave any ShutDownActions in the ShutDownManager at the end of your test, warnings will be issued during the @AfterEach processing. You should remove (and check for correctness) any items that were queued as shutdown items.
• As a temporary bypass when that check was added, the JUnitUtil.clearShutDownManager() method was added. This clears the ShutDownManager without issuing warnings, but also without doing any checks. You should only use this temporarily, hence it's marked as deprecated to flag that.
• When running JUnit tests within the JMRI build infrastructure, a request via InstanceManager.getDefault(..) for a ShutDownManager will get a mock one for testing. A request via InstanceManager.getNullableDefault(..) will get a null. If you want InstanceManager.getNullableDefault(..) to return a manager, you must call JUnitUtil.initShutDownManager() in the @BeforeEach routine to create the mock manager before the getNullableDefault(..) request.

Working with a Timebase

    Timebase clock = InstanceManager.getDefault(jmri.Timebase.class);
    clock.setRun(false);
    clock.setTime(java.time.Instant.EPOCH);  // just a specific time

Working with Listeners

If you want to wait for some specific condition to be true, e.g. receiving a reply object, you can use a waitFor method call which looks like:

    JUnitUtil.waitFor(()->{reply!=null}, "reply didn't arrive");

Waiting for a specific result is fastest and most reliable. If you can't do that for some reason, you can do a short time-based wait:

    JUnitUtil.releaseThread(this);

Note that this should not be used to synchronize with Swing threads. See the Testing Swing Code section for that.

In general, you should not have calls to sleep(), wait() or yield() in your code. Use the JUnitUtil and Jemmy support for those instead.

Working with Threads

Some tests will need to start threads, for example to test signal controls or aspects of layout I/O.

General principles your tests must obey for reliable operation:

• At the end of each test, you need to stop() any threads you started. Doing this in tearDown() can be most reliable, because tearDown runs even if your test method exits due to an error.

  If you're doing multiple tests with threads, you should wait for thread to actually stop before moving on to the next operation. You can do that with a JUnitUtil.waitFor(..) call that waits on some flag in the thread.

• If your thread does any operations at code() that need to happen before you test its operation, you also have to wait for those to complete.

For example, if creating a thread based on AbstractAutomat, you can check the start with:

    AbsractAutomat p = new MyThreadClass();
    p.start();
    JUnitUtil.waitFor(()->{return p.isRunning();}, "logic running");

    p.stop();
    JUnitUtil.waitFor(()->{return !p.isRunning();}, "logic stopped");

Please make sure your unit tests clean up after themselves! They should not leave any threads running. Any threads they start should have either terminated normally by the end of the test (don't let them just time out and crash later during some other test!) or you should add code to terminate them.

You can check whether you've left any threads running by setting the jmri.util.JUnitUtil.checkRemnantThreads environment variable to true, with i.e.

setenv JMRI_OPTIONS -Djmri.util.JUnitUtil.checkRemnantThreads=true

Tools like heap dumps, thread dumps, and the jvisualvm browser can help you see what's being left over by your tests. But they must be used while the JVM is still running, and it usually terminates right after the tests. To prolong the JVM life, add an instance of jmri.util.TestWaitsForever at the end of your PackageList list of tests. TestWaitsForever does what it says on the tin: waits forever, allowing you to look at the state of the JVM. When you're done, you have to kill the test job manually.

Testing I/O

Temporary File Creation in Tests

If you need a temporary file or directory, you can use the JUnit5 TempDir extension to create a file or directory.

The @TempDir annotation can be applied to test class fields of type File or Path.

    import org.junit.jupiter.api.io.TempDir;
    ...
    @TempDir
    protected File folder;

Or as dependency injected values on methods

    import org.junit.jupiter.api.io.TempDir;
    ...
    @Test
    public void testFoo(@TempDir File folder){
       ...
    }

You then reference "folder" in your test code:

        // create a temporary file
        File randomNameFile = folder.newFile();
        // create a temporary directory
        File randomNameDir = folder.newFolder();

JUnit 5 Extensions

1. @org.junit.jupiter.api.io.Tempdir which is used to create temporary folders and files as described previously
2. @org.junit.jupiter.api.Timeout which is used to set a timeout. the @Timeout annotation takes a time as a parameter and can optionally takes units for the given time. The default time unit is seconds, so @Timeout (2) creates a 2 second timeout. The @Timeout annotation may be used on the class or on individual test methods within the class. When applied to the class, the effect is applying the timeout to all @Test annotated methods within the class. described previously

JUnit Rules

    @Rule
    public final TemporaryFolder tempFolder = new TemporaryFolder();

Some standard JUnit rules you might find useful:

• TemporaryFolder - work with temporary files and folders, ensuring they're cleaned up at the end. See the above section for more on using that.
• ExpectedException - handles test methods that are expected to throw exceptions

Timeout - limit duration

    @Rule
    public org.junit.rules.Timeout globalTimeout = org.junit.rules.Timeout.seconds(10);

Note that you can also add a timeout option to an individual test via an argument to the @Test annotation. For example,

    @Test(timeout=2000)

RetryRule - run test several times

For a working example, see java/test/jmri/jmrit/logix/LearnWarrantTest.java

Briefly, you just add the lines

    import jmri.util.junit.rules.RetryRule;

    @Rule
    public RetryRule retryRule = new RetryRule(3);  // allow 3 retries

Tools for Controlling JUnit tests

@Disabled - mark a test to be unconditionally ignored. For example, a test that fails because it isn't fully implemented yet can be marked to be ignored:

        @org.junit.jupiter.api.Disabled("not done yet")
        @jmri.util.junit.annotations.ToDo("Need to create some mock Framistat Objects")
        @Test
        public void notDoneYet() {
            // some code that compiles but doesn't run
        }

You should provide the reason for ignoring this test in the @Disabled argument. @Disabled without an argument will compile, but Jenkins will mark it as an error.

Also note the @jmri.util.junit.annotations.ToDo annotation which indicates that this needs work and provides some more information about what needs to be done. The @ToDo is required when an empty test is provided to ensure that the parent class test doesn't run, but the parent class is applicable to the class under test.

In general, we'd rather have working tests rather than ignored ones, so we track the number that have been ignored in a Jenkins job, see the image to the right.

        @Override
        @jmri.util.junit.annotations.NotApplicable("System X doesn't use Framistat Objects")
        @Test
        public void testFramistatUsage() {}

@DisabledIfSystemProperty(named ="java.awt.headless", matches ="true")

        if (!System.getProperty("jmri.demo", "false").equals("false")) {
            myFrame.setVisible(false);
            myFrame.dispose();
        }

        new Thread(() -> {
            // constructor for d will wait until the dialog is visible
            JDialogOperator d = new JDialogOperator("Foo Warning");
            JButtonOperator bo = new JButtonOperator(d,"OK");
            jmri.util.ThreadingUtil.runOnGUI(() -> {bo.push();});
        }).start();
        showTheDialog();

        myMenuOperator.getTimeouts().setTimeout("JMenuOperator.WaitBeforePopupTimeout", 30L);

    static public void main(String[] args) {
            String[] testCaseName = {"-noloading", LogixTableActionTest.class.getName()};
            junit.swingui.TestRunner.main(testCaseName);
    }