Skip to end of metadata
Go to start of metadata

 

 

This article explains the structure of tests in Magento application and provides unified requirements for implementing PHPUnit-based tests.

Automated tests are built into the Magento framework, allowing developers to easily assess product reliability and performance and to express this assessment using definitive numeric values.

The purpose of automated tests in Magento is to help developers avoid errors when developing new or modifying existing functionality. Requirements, posted in the guides below, have been adopted by the Magento core team. All 3rd-party developers are encouraged to conform to these guidelines in order to provide Magento-related products with similar quality.

Magento also provides the Batch Tests Tool, which uses a script to execute a standardized collections of tests with a single command.

There are different types of automated tests in Magento, some of which are built on top of the PHPUnit testing framework. This includes integration, static and unit testing (performance testing in Magento is not based on PHPUnit).

This guide extends the PHPUnit guide with recommendations and requirements specific to all types of automated tests in Magento built on PHPUnit.

 

 

Tests File Structure Overview

The root folder for all tests is <magento_root>/dev/tests. Test suites and related infrastructure for each type of test are located in folders which correspond to the test types:

  • integration – Integration tests
  • performance – Performance tests (not described in this section)
  • js – Javascript unit tests
  • static – Static tests
  • unit – Unit tests.

The basic structure of each test folder is the following:

Where:

  • tests is located in the directory to which it supplies tests; contains the <test_type> folders
  • <test_type> is the directory that groups tests of the same type as well as the framework to support that type of tests; directory is named for the type of tests it contains
  • testsuite is the directory that contains actual tests

In addition to those mentioned above, each type of test may require additional files and folders; these are described in the documentation for those test types.

Tests File Structure

Utility Classes

Utility classes contain methods which institute logic that is repeated across multiple tests. By using utility methods, developers can avoid the difficulties created by implementing identical (or nearly identical) logic in multiple locations. Utility methods are grouped into classes based on their primary purpose.

Utility classes reside in the Utility sub-directory of the directory containing the tests in which the utility methods are utilized. 

 

 

A utility that is useful for the entire test suite should be placed in the Utility sub-directory of the testsuite directory:

Utility for UnitTest Suite

A utility used for testing a specific module should be placed in the Utility sub-directory of the <module> directory:

Utility for Module Integration Testing

As utility classes are designed to be reused in different places, each class must follow the application autoload rules to be automatically included on demand.

It is preferable to use utility methods as instance methods rather than making them static; this allows utility methods to be more easily tested.

Example

A layout utility for integration tests belonging to the Magento_Core module would be written as follows:

dev/tests/integration/testsuite/Mage/Core/Utility/Layout.php

Using the above layout utility in an integration test would look like this:

dev/tests/integration/testsuite/Mage/DesignEditor/Block/Toolbar/BreadcrumbsTest.php

Running PHPUnit Based Tests

Automated tests in Magento are intended to run from the command line using the phpunit command (assuming the PHPUnit framework is already installed). Use PHPUnit framework v3.6 or later.

PHPUnit Installation

Installation process is documented on the official site

After installation the phpunit command should be available in command line.

Running Tests

Before running tests, change the current directory to the Magento tests root folder:

Change Directory to Magento Tests Root

To run all tests with the default configuration, run the phpunit command from the tests directory:

Run All Tests by Default

When no configuration file is specified, configuration information will automatically be read from phpunit.xml or phpunit.xml.dist (in that order).

To force the use of a specific configuration file, use the configuration switch ("-c"), as described in the PHPUnit framework TextUI switches:

Run All Tests with Default Configuration File Specified

Running One or a Group of Test Cases

To run a particular test case, specify the path to its file relative to the current directory (tests framework root), for example:

Run a Single Test Case by Class Name
Run a Particular Test Case

If a folder is specified, PHPUnit will find and run all test cases from that folder:

Run All Test Cases in a Folder

Batch Tests Tool

The Batch Tests Tool is a script that automatically runs a collection of test suites. Usage:

Default Usage of Batch Tool

By default, the Batch Tests Tool runs all available tests, except those which take take long time by default: static "Legacy", integration "integrity" suite and functional tests, etc... To run all tests, use "--type=all" argument:

Executing All Tests Using Batch Tool

More granular control is available by specifying other types which include (but are not limited to): "unit", "integration", "static", "integrity", "legacy". But some of the types may execute not entire test suite for the same reason as described above – long execution time. Add "-all" suffix to run all tests of this type, for example, "--type=static-all".

Whenever a not supported type is specified, the program will terminate and list all available types.

Following execution, the Batch Tests Tool produces a detailed report of the tests run and their results.

Run Batch Tests Tool

Running Tests Using PhpStorm

PHPUnit can be run and unit tests can be executed from PhpStorm. Perform the following steps to setup PHPUnit in your PhpStorm:

1. Modify your PHP and PHPUnit settings:

  • Open Settings by pressing Ctrl+Alt+S or clicking  on the main toolbar.
  • Under Project Settings select PHP.
  • Make sure you set PHP language level and Interpreter. For Magento 2 minimum PHP version is PHP 5.4.
  • Under PHP select PHPUnit.
  • Select the Use configuration file check box and enter the absolute path to the dev/tests/unit/phpunit.xml.dist file.
  • Select the Use bootstrap file check box and enter the absolute path to the dev/tests/unit/framework/bootstrap.php file.

2. Create a PHPUnit Run Configuration, using the Run > Edit Configurations menu, and specify  the dev/tests/unit/testsuite as directory for Test Runner.

To run the unit tests using PHPUnit, select your Run Configuration and click the Run green right arrow icon next to the Run Configuration dropdown menu located at the top of the IDE. The test results will appear in a panel located at the bottom of the IDE.

An optional PHPUnit code coverage plugin is available from the Plugins repository. Perform the following steps to setup PHPUnit code coverage in your IDE.

To install the Code Coverage Plugin:

  1. Open Settings and select Plugins
  2. Click the Browse Repositories... button
  3. Right-click PHPUnit code coverage and select Download and Install
  4. Restart PhpStorm

Configure the Code Coverage Plugin

  1. Open Settings and select PHPUnit Coverage
  2. Specify the absolute path to your clover.xml file in the Clover xml location field
  3. Optionally select different colors for Covered and Uncovered code
  4. Select Line for Highlight

To run the unit tests using PHPUnit with code coverage, select your Run Configuration and click the Run with Coverage green right arrow icon next to the Debug icon located at the top of the IDE. The test results will appear in a panel located at the bottom of the IDE.

When you open PHP files whose code has been executed by PHPUnit, the Covered and Uncovered code will appear using the colors specified in the Plugin's configuration.

Customizing Test Bootstrap Settings

Magento automated tests are supplied with the phpunit.xml.dist file, which out of the box provides default bootstrap settings for running tests. Users can copy it as phpunit.xml and edit it as detailed in The XML Configuration File.

If the file is copied with any name other than phpunit.xml, the file must be specified using the "-c" switch:

Run All Tests With Custom Bootstrap Configuration

Customizing Code Coverage Filters

The PHPUnit framework uses the <filter> xml section 

Run All Tests for Default Base Folder

Modify or add to the existing <directory> nodes to customize the list of folders to be scanned. For example, to scan the Magento/Checkout folder:

Run All Tests for Mage/Checkout Folder Only

Generating Coverage Report

Code coverage report generation can be pre-configured for Magento automated tests. By default, this report generation is disabled because of the potential impact on environment and performance requirements.

The report generation feature requires the xDebug PHP extension to be installed.

To enable code coverage report generation, uncomment the <logging> section in the XML configuration file. The report output locations are specified in this section as <log> nodes:

Specify Code Coverage Report Location
Icon

Generating the code coverage report is a memory-intensive operation. It can cause "PHP Fatal error: Allowed memory size of ... bytes exhausted...".

There are two ways to avoid fatal errors caused by memory size being exhausted:

  1. Increase the "memory_limit" php.ini directive (up to a maximum of 1024M).
  2. Limit the scope of the code coverage filters in the filter/whitelist/directory:

    Limit Scope of Code Coverage Report

Requirements for Implementing Tests

Icon

These are internal requirements, published to provide external developers with a general coding style and testing quality.

 

Tests in Magento are intended to be read and updated. There are certain formal requirements for writing PHPUnit-based tests that help make all tests easier to maintain.

1. Naming Test Case Classes

Assign a name to each test case class that corresponds to the original class by adding "Test" to the end. This helps facilitate navigation and explicitly designates that this class is a test case.

Name a Test Case

2. Placing setUp() and tearDown() Methods

If the test case includes setUpBeforeClass(), tearDownAfterClass(), setUp(), tearDown(), or any combination of these methods, place them before other methods in the class. This helps identify preconditions and post-conditions for all tests in this test case.

Place Test Setup Methods First

3. Naming Data Providers

Data provider method names must end with the phrase DataProvider. Name data provider methods using the pattern <name>DataProvider, where <name> can be:

  • the original method name
  • a unique method name (if one data provider is used in different test methods, for example)
Name Data Providers

4. Noting Skipped or Incomplete Tests

Always note the reason a test is incomplete or skipped in the argument of the respective method.

Mark a test incomplete in case, when it cannot be performed because of some bug or not yet implemented code.

Note Reason for an Incomplete Test

Mark test skipped in case, when it can't be performed at all (e.g., test related to MSSQL can't be performed, if tests are run on MySQL).

5. Calling Fake Abstract Classes in Magento

PHPUnit provides a built-in mechanism for testing PHP abstract classes:

The getMockForAbstractClass() method returns a mock object for an abstract class. All abstract methods of the given abstract class are mocked. This allows for testing the concrete methods of an abstract class.

http://www.phpunit.de/manual/current/en/test-doubles.html

In Magento there are classes that are declared as abstract, but in fact do not contain any abstract methods. A test for such a class would look like the following example:

Fake Abstract Class Test

Recommendations for Implementing Tests

Creating Test Case Class Skeletons

To generate a test case class that conforms to all mentioned requirements, use the standard phpunit command option "skeleton-test". Specify the class name for which the test case will be generated. The include path, specified in the configuration file (phpunit.xml or similar) will be determined automatically:

Generate a Test Skeleton

The command will return the location of a new file containing the test case class.

Organizing Test Methods

Arrange test methods in the same order as the public methods in the original class. This allows the test case and the original class to be read simultaneously when reviewing test implementation.

Original Methods
Test Case Methods

Naming Test Methods

A typical name for a test method would be the name of the original method with the prefix "test" added. The "test" prefix is also required by the PHPUnit framework to distinguish tests from other methods.

If multiple tests are required to fully test a particular method, name each related test as described above, then add a word to the end that clearly describes the the distinguishing characteristic of the specific test case:

Name Additional Specific Tests

Sometimes one method can be tested in conjunction with another. In this case, the test would have a name that identifies both methods being tested. Typically such methods are pairs of a setter and getter:

Name Combined Test for Setter and Getter

If the original method name is not obvious from the test method name, mention it in docblock comment (as described in @covers annotation). Reasons a method being tested might be ambiguous are:

  1. The test method combines testing of several original methods:

    Identify Original Methods for Combined Tests
  2. The test method tests an abstract class's method(s) through its descendant:

    Identify Test Method for Method of Abstract Class

Covering all Methods

Cover, or at least mention, in the test case all public methods of the original class. This clearly identifies any incomplete methods of the original class.

Original Methods
Test Case Methods
Icon

Mentioning a method in @covers notation implies that it is fully covered by the test. Use this carefully to avoid spoiling code coverage statistics.

Declaring Data Providers

Declare each data provider method immediately after the respective test method (as seen in data providers in PHPUnit documentation) . This provides context for the data provider when reading the class from top to bottom.

Using Compact Docblock Comments

Integration tests are intended to be read and modified frequently, and their API is not intended to be published in any documentation. Therefore, keep docblock comments minimal and include only required information, such as:

  • any docblock required by the PHPUnit framework
  • type hinting for class attributes
  • which methods of which class are covered by this test (if not evident from the method name)
Excessive Docblock
Compact Docblock

Don't omit docblock for auxiliary methods (methods not implied by the PHPUnit framework):

Document "Unusual" Things

Using Brief Assertion Comments

The PHPUnit framework provides numerous public methods for various kinds of assertions, which speak for themselves. If there is a simple and a well-defined assertion, it requires no additional comment.

  • This assertion doesn't need a comment:

    Example: Self-Sufficient Assertion
  • This comment is not useful, and just obstructs reading:

    Example: Excessive Comment for Assertion
  • Here it is not obvious which types of values are in the $group['block1'] and $blocks[0] variables; a comment may clarify this:

    Example: A Complicated Assertion with a Useful Comment

Unit Tests: Mocking Constructor Parameters Using Object Manager Helper

In Magento 2 many block and model classes declare excessive dependencies in constructors (Magento 2 uses constructor dependency injection). In order to cover such classes with unit tests, a developer needs to create mocks for all constructor parameters manually, which might be time-consuming.

To facilitate this routine, you can use \Magento\TestFramework\Helper\ObjectManager, which provides methods that automatically create mocks for all required dependencies (you can still provide your custom mocks if needed), and then instantiate testing object by passing these mocks to a class constructor.

For information about ObjectManager helper class usage, please refer to the Object Manager Helper article. 

PHPUnit Compatibility Tips (v3.6)

Some of the changes in PHPUnit 3.6 cause tests which ran effectively on earlier versions to fail unexpectedly. To ensure the compatibility of tests with PHPUnit 3.6, consider the following recommendations.

Testing Exceptions

In PHPUnit 3.6 exception testing has been changed:

Note
You should be as specific as possible when testing exceptions. Testing for classes that are too generic might lead to undesirable side-effects. Accordingly, testing for the Exception class with @expectedException or setExpectedException() is no longer permitted.

http://www.phpunit.de/manual/3.6/en/writing-tests-for-phpunit.html

The following examples demonstrate how tests (and the code being tested) must be modified to be compatible:

PHPUnit 3.5 and lower, PHPUnit 3.7

PHPUnit 3.6

Mocked Method Callback

PHPUnit provides the ability to mock a method with a callback. However, as soon as a mocked method accepts arguments by reference PHPUnit 3.6 (3.6.10 in particular) produces the following error:

Parameter 1 to {closure}() expected to be a reference, value given

The only way to avoid this error is to re-factor the code so it doesn't pass arguments by reference. For example: