Automated Screenshot-Based Testing
Imagine you have hundreds of static pages and hundreds of product pages and a comprehensive system which is able to change things on any of them in any second. This system involves the software components such as data integration or data cleanup modules, and people, such as administrators or developers. Some of the actions generated by them are trackable, but not all. Even if the actions are trackable, it is not easy to distinguish what changes affected the storefront and what aren’t.
I faced this problem last year. We were preparing the system for a launch, and we needed to know about all processes in the system. Especially those which affect the storefront. Especially if these processes lead to removing or changing things there.
Before and after another deployment some pages and components are not expected to be changed, but things happen and we need to get prepared and be aware of any break at the earliest.
This situation was compounded by the fact that we had a distributed network of content administrators who were taking their first steps in content management. Having such setup, we were facing the content issues that require us to put significant efforts into finding a root cause. Lack of SAP hybris auditing capabilities (who did what and when) added a decent level of complexity.
In order to keep control on the changes, I created a quick solution to monitor them. My script created a set of screenshots and compared them with the previous set. The report showed the differences. Having such reports, we were able to detect unexpected changes in the website content. It helped us to detect the bugs in the code as well.
There are three steps:
driver.execute_script is used to execute Javascript code in the browser. It helped me to hide some blocks before getting a screenshot. The website uses the menu fixed on top when scrolled. Without hiding it, the menu will be repeated in the screenshot as many times as the number of screens in it.
The JSP file for this controller is simple:
You need to apply the impexes for this page:
However, this tool requires that the images must have the same dimensions. If your component or page was changed, the size will be different. You need to resize the images to the dimensions of the larger image.
I used bash for the makeSameSizeImages.sh script. It was not the best solution, but a quick one that works.
Having two same-size screenshots of the same page created by the same tool with the same browser, we can assume that the result of comparison of the them will be absolutely predictable – no differences if no changes were made with the components of these pages.
However, the chrome rendering engine doesn’t create absolutely identical screenshots even if all components are identical. Unfortunately, there is no pixel-wise accuracy for the images. Look at the banners in the right column. The system thought that they are different a bit, but it is not so.
These images look identical:
and
But the image comparison shows that there is a difference in the icon:
This icon is added as a vector image on top of the raster image, and chrome turns antialiasing on, and it is a bit random.
We need to ignore these changes by adding
as a parameter:
You can see, that the banners are not marked as different anymore.
That is why I compare component pages, not only the whole pages. Components, such as banners in this case or news feeds, are compared separately being put on the page in isolation from other components.
You can see that the bottom part is marked in red:
This is so because our HTML works with relative positioning rather than the absolute positioning of the objects. One change in the middle of the list, and all items of this list below the item will be considered as different, while they are just shifted. Comparision of components rather than pages makes the situation easier to resolve.
Architecture
- Creating screenshots (image files)
- Includes SSO authentication
- Supports batch mode
- Comparing with the previous state
- Supports fuzziness to turn a blind eye to rendering-related differences
- Creating a report
- Major differences are clearly seen
- Minor differences can be filtered out
- WCMS pages
- Public pages
- Restricted access pages
- Component pages
- Per type: a page per component type. Each page contains all the component instances of the component type
- Per component: a page per component instance. Each page contains only one component instance.
- Current language (session)
- Current country (session)
- Current user (session)
Creating Screenshots
I used Selenium Server 3.7 + Chromedriver + Python 3 + Selenium for Python for the purpose.- pip3 install selenium
- pip3 install pyvirtualdisplay
- brew install chromedriver (for MacOS)
from pyvirtualdisplay import Display
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
display = Display(visible=0, size=(1700, 800))
display.start()
driver = webdriver.Chrome(service_args=["--verbose --log-path=chrome.log"])
driver.set_window_size(1700, 800)
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element_by_name("q")
elem.clear()
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
driver.close()
display.stop()
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
display = Display(visible=0, size=(1700, 800))
display.start()
driver = webdriver.Chrome(service_args=["--verbose --log-path=chrome.log"])
driver.set_window_size(1700, 800)
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element_by_name("q")
elem.clear()
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
driver.close()
display.stop()
Navigating and loading the page for screenshotting
In order to load a page, you need to use the syntax like the following:driver.get("http://www.python.org")
My script iterates over the pages and loads them one by one in the virtual browser (Supported by chromedriver).
Authentication. More than half of the pages to test required authentication. The website I was working is integrated with OKTA as an SSO provide. The script submits a form to get an access to these pages.
Responsive. The website supports responsive mode that means that different devices and resolutions are supposed to be processed differently, a special set of templates will be used etc. So there was a separate run of the process with the mobile mode on.
The following code loads the pages and launcher the screeenshotter, the results are placed into the HTML files and image folder. The URL depends on the country and language, that is why these components are in the equation.
Creating a screenshot
The script creates the image files (screenshots) in the folder with the name unique to the process.
driver.execute_script is used to execute Javascript code in the browser. It helped me to hide some blocks before getting a screenshot. The website uses the menu fixed on top when scrolled. Without hiding it, the menu will be repeated in the screenshot as many times as the number of screens in it.
SAP hybris Components page
Creating full-page screenshots is not a solution if it contains many dynamic blocks: your screenshots will be too different from each other to detect changes reliably. The solution is in tracking changes with the page fragments – WCMS components. However, hybris OOTB doesn’t provide the way of rendering the component separately from the page it is contained in. There is a customization: the page controller which renders the components taken from URL parameter.@RequestMapping(value = "/_wcmscomponents")
public class GetAllComponentsController extends AbstractPageController
{
@Autowired
private ModelService modelService;
@Autowired
private TypeService typeService;
@Autowired
private CMSComponentService cmsComponentService;
private Set showAllComponents(final Collection allPages,
final Predicate filter)
{
return allPages.stream()
.map(AbstractPageModel::getContentSlots)
.flatMap(Collection::stream)
.map(ContentSlotForPageModel::getContentSlot)
.map(ContentSlotModel::getCmsComponents)
.flatMap(Collection::stream)
.filter(SimpleCMSComponentModel.class::isInstance)
.filter(filter)
.collect(Collectors.toSet());
}
@RequestMapping(value = "/{uid}", method = RequestMethod.GET)
public String getByUid(final Model model, @PathVariable("uid") final String uid) throws CMSItemNotFoundException
{
final SimpleCMSComponentModel simpleCMSComponent = cmsComponentService.getSimpleCMSComponent(uid);
final AbstractPageModel page = getAvailableComponentsPage();
final ContentSlotModel bodySlot = getBodySlot(page);
addComponents(bodySlot, Collections.singletonList(simpleCMSComponent));
storeCmsPageInModel(model, page);
return getViewForPage(page);
}
private AbstractPageModel getAvailableComponentsPage() throws CMSItemNotFoundException
{
return getContentPageForLabelOrId("availableComponents");
}
private ContentSlotModel getBodySlot(final AbstractPageModel page) throws CMSItemNotFoundException
{
return getCmsPageService().getContentSlotForPage(page, "BodySection").getContentSlot();
}
private void addComponents(final ContentSlotModel slot, final List availableComponents)
{
slot.setCmsComponents(availableComponents);
modelService.save(slot);
}
public class GetAllComponentsController extends AbstractPageController
{
@Autowired
private ModelService modelService;
@Autowired
private TypeService typeService;
@Autowired
private CMSComponentService cmsComponentService;
private Set showAllComponents(final Collection allPages,
final Predicate filter)
{
return allPages.stream()
.map(AbstractPageModel::getContentSlots)
.flatMap(Collection::stream)
.map(ContentSlotForPageModel::getContentSlot)
.map(ContentSlotModel::getCmsComponents)
.flatMap(Collection::stream)
.filter(SimpleCMSComponentModel.class::isInstance)
.filter(filter)
.collect(Collectors.toSet());
}
@RequestMapping(value = "/{uid}", method = RequestMethod.GET)
public String getByUid(final Model model, @PathVariable("uid") final String uid) throws CMSItemNotFoundException
{
final SimpleCMSComponentModel simpleCMSComponent = cmsComponentService.getSimpleCMSComponent(uid);
final AbstractPageModel page = getAvailableComponentsPage();
final ContentSlotModel bodySlot = getBodySlot(page);
addComponents(bodySlot, Collections.singletonList(simpleCMSComponent));
storeCmsPageInModel(model, page);
return getViewForPage(page);
}
private AbstractPageModel getAvailableComponentsPage() throws CMSItemNotFoundException
{
return getContentPageForLabelOrId("availableComponents");
}
private ContentSlotModel getBodySlot(final AbstractPageModel page) throws CMSItemNotFoundException
{
return getCmsPageService().getContentSlotForPage(page, "BodySection").getContentSlot();
}
private void addComponents(final ContentSlotModel slot, final List availableComponents)
{
slot.setCmsComponents(availableComponents);
modelService.save(slot);
}
You need to apply the impexes for this page:
INSERT_UPDATE ContentPage; $contentCV[unique = true]; uid[unique = true]; name; masterTemplate(uid, $contentCV); label; label; defaultPage[default = 'true']; approvalStatus(code)[default = 'approved']; homepage[default = 'false'] ; ; availableComponents ; Available Components Page ; AvailableComponentsPageTemplate ; availableComponents ; available-components INSERT_UPDATE ContentSlot; $contentCV[unique = true]; uid[unique = true]; name; active; cmsComponents(&componentRef) ; ; AvailableComponentsBodySlot ; AvailableComponents Body Slot ; true ; INSERT_UPDATE ContentSlotForPage; $contentCV[unique = true]; uid[unique = true]; position[unique = true]; page(uid, $contentCV)[unique = true][default = 'availableComponents']; contentSlot(uid, $contentCV)[unique = true] ; ; BodySection-AvailableComponents ; BodySection ; ; AvailableComponentsBodySlotAs a result, you will be able to render the component in isolation. Create an URL for each component and you will get a list for the screenshotting engine. For the my case, the scripts created the following structure:
- screenshots_20180325_140000/
- cmspages/
- desktop
- DE
- EN
- company
- about-us.html
- images/
- about-us.png
- our-products.html
- images/
- our-products.png
- company
- DE
- company
- about-us.html
- images/
- about-us.png
- our-products.html
- images/
- our-products.png
- company
- EN
- DE
- mobile
- DE
- EN
- …
- EN
- DE
- desktop
- components/
- desktop
- DE
- EN
- AboutCompanySimpleParagraphComponent.html
- OurProductsParagraphComponent.html
- images/
- AboutCompanySimpleParagraphComponent.png
- OurProductsParagraphComponent.png
- DE
- …
- EN
- DE
- desktop
- cmspages/
Comparing the screenshot with the previous state of the page
Each page and each component have their own page and a separate screenshot image file. It is enough for manual check, but when you have hundreds of pages, many language and country versions of the pages, it is not a solution. You need to track the changes automatically. I use ImageMagick for this purpose. For each image file, my script finds a counterpart in the folder created at the previous run, and compares the files using the ImageMagick compare tool:compare -metric AE $CURDIR/$TARGET/$file1 $CURDIR/$TARGET/$file2 $CURDIR/$TARGET/$file
$CURDIR/makeSameSizeImages.sh $CURDIR/$SOURCE1/$file $CURDIR/$SOURCE2/$file $CURDIR/$TARGET/$file1 $CURDIR/$TARGET/$file2
#!/bin/bash
SIZE=`identify $1 | cut -d " " -f 3`
WIDTH1=`echo $SIZE | perl -npe "s/^(.*?)x(.*?)$/\1/g"`
HEIGHT1=`echo $SIZE | perl -npe "s/^(.*?)x(.*?)$/\2/g"`
SIZE=`identify $2 | cut -d " " -f 3`
WIDTH2=`echo $SIZE | perl -npe "s/^(.*?)x(.*?)$/\1/g"`
HEIGHT2=`echo $SIZE | perl -npe "s/^(.*?)x(.*?)$/\2/g"`
MAX=$WIDTH2
if [ $WIDTH1 -gt $WIDTH2 ];
then
MAX=$WIDTH1
fi
if [ $WIDTH2 -gt $WIDTH1 ];
then
MAX=$WIDTH2
fi
MAXWIDTH=$MAX
MAX=$HEIGHT2
if [ $HEIGHT1 -gt $HEIGHT2 ];
then
MAX=$HEIGHT1
fi
if [ $HEIGHT2 -gt $HEIGHT1 ];
then
MAX=$HEIGHT2
fi
MAXHEIGHT=$MAX
echo "[RESIZE] width=" $WIDTH1 $WIDTH2
echo "[RESIZE] height=" $HEIGHT1 $HEIGHT2
echo "[RESIZE] max = " $MAXWIDTH $MAXHEIGHT
if [ "$WIDTH1 x $HEIGHT1" != "$MAXWIDTH x $MAXHEIGHT" ];
then
echo "first image should be resized";
convert -resize ${MAXWIDTH}x${MAXHEIGHT} -extent ${MAXWIDTH}x${MAXHEIGHT} -gravity northwest $1 $3
else
cp $1 $3
fi
if [ "$WIDTH2 x $HEIGHT2" != "$MAXWIDTH x $MAXHEIGHT" ];
then
echo "second image should be resized";
convert -resize ${MAXWIDTH}x${MAXHEIGHT} -extent ${MAXWIDTH}x${MAXHEIGHT} -gravity northwest $2 $4
else
cp $2 $4
fi
SIZE=`identify $1 | cut -d " " -f 3`
WIDTH1=`echo $SIZE | perl -npe "s/^(.*?)x(.*?)$/\1/g"`
HEIGHT1=`echo $SIZE | perl -npe "s/^(.*?)x(.*?)$/\2/g"`
SIZE=`identify $2 | cut -d " " -f 3`
WIDTH2=`echo $SIZE | perl -npe "s/^(.*?)x(.*?)$/\1/g"`
HEIGHT2=`echo $SIZE | perl -npe "s/^(.*?)x(.*?)$/\2/g"`
MAX=$WIDTH2
if [ $WIDTH1 -gt $WIDTH2 ];
then
MAX=$WIDTH1
fi
if [ $WIDTH2 -gt $WIDTH1 ];
then
MAX=$WIDTH2
fi
MAXWIDTH=$MAX
MAX=$HEIGHT2
if [ $HEIGHT1 -gt $HEIGHT2 ];
then
MAX=$HEIGHT1
fi
if [ $HEIGHT2 -gt $HEIGHT1 ];
then
MAX=$HEIGHT2
fi
MAXHEIGHT=$MAX
echo "[RESIZE] width=" $WIDTH1 $WIDTH2
echo "[RESIZE] height=" $HEIGHT1 $HEIGHT2
echo "[RESIZE] max = " $MAXWIDTH $MAXHEIGHT
if [ "$WIDTH1 x $HEIGHT1" != "$MAXWIDTH x $MAXHEIGHT" ];
then
echo "first image should be resized";
convert -resize ${MAXWIDTH}x${MAXHEIGHT} -extent ${MAXWIDTH}x${MAXHEIGHT} -gravity northwest $1 $3
else
cp $1 $3
fi
if [ "$WIDTH2 x $HEIGHT2" != "$MAXWIDTH x $MAXHEIGHT" ];
then
echo "second image should be resized";
convert -resize ${MAXWIDTH}x${MAXHEIGHT} -extent ${MAXWIDTH}x${MAXHEIGHT} -gravity northwest $2 $4
else
cp $2 $4
fi
and
But the image comparison shows that there is a difference in the icon:
This icon is added as a vector image on top of the raster image, and chrome turns antialiasing on, and it is a bit random.
We need to ignore these changes by adding
-fuzz 5%
You can see, that the banners are not marked as different anymore.
That is why I compare component pages, not only the whole pages. Components, such as banners in this case or news feeds, are compared separately being put on the page in isolation from other components.
You can see that the bottom part is marked in red:
This is so because our HTML works with relative positioning rather than the absolute positioning of the objects. One change in the middle of the list, and all items of this list below the item will be considered as different, while they are just shifted. Comparision of components rather than pages makes the situation easier to resolve.
Technologies and tools used
- Pyhton 3
- Selenium Server 3.7.1
- Chromedriver 2.33
- Bash
- Imagemagick 6.7
