This article explains the structure of tests in Magento application and provides unified requirements for implementing PHPUnit-based tests.
Automated tests are built into the Magento framework, allowing developers to easily assess product reliability and performance and to express this assessment using definitive numeric values.
The purpose of automated tests in Magento is to help developers avoid errors when developing new or modifying existing functionality. Requirements, posted in the guides below, have been adopted by the Magento core team. All 3rd-party developers are encouraged to conform to these guidelines in order to provide Magento-related products with similar quality.
Magento also provides the Batch Tests Tool, which uses a script to execute a standardized collections of tests with a single command.
There are different types of automated tests in Magento, some of which are built on top of the PHPUnit testing framework. This includes integration, static and unit testing (performance testing in Magento is not based on PHPUnit).
This guide extends the PHPUnit guide with recommendations and requirements specific to all types of automated tests in Magento built on PHPUnit.
Table of Contents:
- Tests File Structure Overview
- Running PHPUnit Based Tests
- Customizing Test Bootstrap Settings
- Requirements for Implementing Tests
- Recommendations for Implementing Tests
- PHPUnit Compatibility Tips (v3.6)
Tests File Structure Overview
The root folder for all tests is
<magento_root>/dev/tests. Test suites and related infrastructure for each type of test are located in folders which correspond to the test types:
integration– Integration tests
performance– Performance tests (not described in this section)
static– Static tests
unit– Unit tests.
The basic structure of each test folder is the following:
testsis located in the directory to which it supplies tests; contains the
<test_type>is the directory that groups tests of the same type as well as the framework to support that type of tests; directory is named for the type of tests it contains
testsuiteis the directory that contains actual tests
In addition to those mentioned above, each type of test may require additional files and folders; these are described in the documentation for those test types.
Utility classes contain methods which institute logic that is repeated across multiple tests. By using utility methods, developers can avoid the difficulties created by implementing identical (or nearly identical) logic in multiple locations. Utility methods are grouped into classes based on their primary purpose.
Utility classes reside in the
Utility sub-directory of the directory containing the tests in which the utility methods are utilized.
A utility that is useful for the entire test suite should be placed in the
Utility sub-directory of the
A utility used for testing a specific module should be placed in the
Utility sub-directory of the
As utility classes are designed to be reused in different places, each class must follow the application autoload rules to be automatically included on demand.
It is preferable to use utility methods as instance methods rather than making them
static; this allows utility methods to be more easily tested.
A layout utility for integration tests belonging to the
Magento_Core module would be written as follows:
Using the above layout utility in an integration test would look like this:
Running PHPUnit Based Tests
Automated tests in Magento are intended to run from the command line using the
phpunit command (assuming the PHPUnit framework is already installed). Use PHPUnit framework v3.6 or later.
Installation process is documented on the official site
After installation the
phpunit command should be available in command line.
Before running tests, change the current directory to the Magento tests root folder:
To run all tests with the default configuration, run the
phpunit command from the tests directory:
When no configuration file is specified, configuration information will automatically be read from
phpunit.xml.dist (in that order).
To force the use of a specific configuration file, use the configuration switch ("
-c"), as described in the PHPUnit framework TextUI switches:
Running One or a Group of Test Cases
To run a particular test case, specify the path to its file relative to the current directory (tests framework root), for example:
If a folder is specified, PHPUnit will find and run all test cases from that folder:
Batch Tests Tool
The Batch Tests Tool is a script that automatically runs a collection of test suites. Usage:
By default, the Batch Tests Tool runs all available tests, except those which take take long time by default: static "Legacy", integration "integrity" suite and functional tests, etc... To run all tests, use "
More granular control is available by specifying other types which include (but are not limited to): "
legacy". But some of the types may execute not entire test suite for the same reason as described above – long execution time. Add "
-all" suffix to run all tests of this type, for example, "
Whenever a not supported type is specified, the program will terminate and list all available types.
Following execution, the Batch Tests Tool produces a detailed report of the tests run and their results.
Running Tests Using PhpStorm
PHPUnit can be run and unit tests can be executed from PhpStorm. Perform the following steps to setup PHPUnit in your PhpStorm:
1. Modify your PHP and PHPUnit settings:
- Open Settings by pressing Ctrl+Alt+S or clicking on the main toolbar.
- Under Project Settings select PHP.
- Make sure you set PHP language level and Interpreter. For Magento 2 minimum PHP version is PHP 5.4.
- Under PHP select PHPUnit.
- Select the Use configuration file check box and enter the absolute path to the
- Select the Use bootstrap file check box and enter the absolute path to the
2. Create a PHPUnit Run Configuration, using the Run > Edit Configurations menu, and specify the
dev/tests/unit/testsuite as directory for Test Runner.
To run the unit tests using PHPUnit, select your Run Configuration and click the
Run green right arrow icon next to the Run Configuration dropdown menu located at the top of the IDE. The test results will appear in a panel located at the bottom of the IDE.
PHPUnit code coverage plugin is available from the
Plugins repository. Perform the following steps to setup
PHPUnit code coverage in your IDE.
To install the Code Coverage Plugin:
- Click the
PHPUnit code coverageand select
Download and Install
- Restart PhpStorm
Configure the Code Coverage Plugin
- Specify the absolute path to your
clover.xmlfile in the
Clover xml locationfield
- Optionally select different colors for
To run the unit tests using PHPUnit with code coverage, select your Run Configuration and click the
Run with Coverage green right arrow icon next to the
Debug icon located at the top of the IDE. The test results will appear in a panel located at the bottom of the IDE.
When you open PHP files whose code has been executed by PHPUnit, the
Uncovered code will appear using the colors specified in the Plugin's configuration.
Customizing Test Bootstrap Settings
Magento automated tests are supplied with the
phpunit.xml.dist file, which out of the box provides default bootstrap settings for running tests. Users can copy it as
phpunit.xml and edit it as detailed in The XML Configuration File.
If the file is copied with any name other than
phpunit.xml, the file must be specified using the "
Customizing Code Coverage Filters
The PHPUnit framework uses the
<filter> xml section
Modify or add to the existing
<directory> nodes to customize the list of folders to be scanned. For example, to scan the
Generating Coverage Report
Code coverage report generation can be pre-configured for Magento automated tests. By default, this report generation is disabled because of the potential impact on environment and performance requirements.
The report generation feature requires the xDebug PHP extension to be installed.
To enable code coverage report generation, uncomment the
<logging> section in the XML configuration file. The report output locations are specified in this section as
There are two ways to avoid fatal errors caused by memory size being exhausted:
- Increase the "
php.inidirective (up to a maximum of 1024M).
Limit the scope of the code coverage filters in the
Requirements for Implementing Tests
Tests in Magento are intended to be read and updated. There are certain formal requirements for writing PHPUnit-based tests that help make all tests easier to maintain.
1. Naming Test Case Classes
Assign a name to each test case class that corresponds to the original class by adding "Test" to the end. This helps facilitate navigation and explicitly designates that this class is a test case.
2. Placing setUp() and tearDown() Methods
If the test case includes
tearDown(), or any combination of these methods, place them before other methods in the class. This helps identify preconditions and post-conditions for all tests in this test case.
3. Naming Data Providers
Data provider method names must end with the phrase
DataProvider. Name data provider methods using the pattern
<name> can be:
- the original method name
- a unique method name (if one data provider is used in different test methods, for example)
4. Noting Skipped or Incomplete Tests
Always note the reason a test is incomplete or skipped in the argument of the respective method.
Mark a test incomplete in case, when it cannot be performed because of some bug or not yet implemented code.
Mark test skipped in case, when it can't be performed at all (e.g., test related to MSSQL can't be performed, if tests are run on MySQL).
5. Calling Fake Abstract Classes in Magento
PHPUnit provides a built-in mechanism for testing PHP abstract classes:
getMockForAbstractClass()method returns a mock object for an abstract class. All abstract methods of the given abstract class are mocked. This allows for testing the concrete methods of an abstract class.
In Magento there are classes that are declared as abstract, but in fact do not contain any abstract methods. A test for such a class would look like the following example:
Recommendations for Implementing Tests
Creating Test Case Class Skeletons
To generate a test case class that conforms to all mentioned requirements, use the standard phpunit command option "skeleton-test". Specify the class name for which the test case will be generated. The include path, specified in the configuration file (
phpunit.xml or similar) will be determined automatically:
The command will return the location of a new file containing the test case class.
Organizing Test Methods
Arrange test methods in the same order as the public methods in the original class. This allows the test case and the original class to be read simultaneously when reviewing test implementation.
Naming Test Methods
A typical name for a test method would be the name of the original method with the prefix "test" added. The "test" prefix is also required by the PHPUnit framework to distinguish tests from other methods.
If multiple tests are required to fully test a particular method, name each related test as described above, then add a word to the end that clearly describes the the distinguishing characteristic of the specific test case:
Sometimes one method can be tested in conjunction with another. In this case, the test would have a name that identifies both methods being tested. Typically such methods are pairs of a setter and getter:
If the original method name is not obvious from the test method name, mention it in docblock comment (as described in @covers annotation). Reasons a method being tested might be ambiguous are:
The test method combines testing of several original methods:
The test method tests an abstract class's method(s) through its descendant:
Covering all Methods
Cover, or at least mention, in the test case all public methods of the original class. This clearly identifies any incomplete methods of the original class.
Declaring Data Providers
Declare each data provider method immediately after the respective test method (as seen in data providers in PHPUnit documentation) . This provides context for the data provider when reading the class from top to bottom.
Using Compact Docblock Comments
Integration tests are intended to be read and modified frequently, and their API is not intended to be published in any documentation. Therefore, keep docblock comments minimal and include only required information, such as:
- any docblock required by the PHPUnit framework
- type hinting for class attributes
- which methods of which class are covered by this test (if not evident from the method name)
Don't omit docblock for auxiliary methods (methods not implied by the PHPUnit framework):
Using Brief Assertion Comments
The PHPUnit framework provides numerous public methods for various kinds of assertions, which speak for themselves. If there is a simple and a well-defined assertion, it requires no additional comment.
This assertion doesn't need a comment:
This comment is not useful, and just obstructs reading:
Here it is not obvious which types of values are in the
variables; a comment may clarify this:
PHPUnit Compatibility Tips (v3.6)
Some of the changes in PHPUnit 3.6 cause tests which ran effectively on earlier versions to fail unexpectedly. To ensure the compatibility of tests with PHPUnit 3.6, consider the following recommendations.
In PHPUnit 3.6 exception testing has been changed:
You should be as specific as possible when testing exceptions. Testing for classes that are too generic might lead to undesirable side-effects. Accordingly, testing for the Exception class with @expectedException or setExpectedException() is no longer permitted.
The following examples demonstrate how tests (and the code being tested) must be modified to be compatible:
PHPUnit 3.5 and lower, PHPUnit 3.7
Mocked Method Callback
The only way to avoid this error is to re-factor the code so it doesn't pass arguments by reference. For example: