Forem: Tomas Norre Mikkelsen

Running TYPO3 11 with ddev

Tomas Norre Mikkelsen — Sat, 02 Jan 2021 12:43:26 +0000

I recently started a new Project with TYPO3 11 and ddev, there a two things your should be aware of.

Make sure to use ddev 1.16.5 or never.
Add the default .htaccess to you public-folder, otherwise the backend Route will not work.

My setup is as you see, with apacehe-fpm.

When Running with nginx-fpm no additional configuration is needed besides ddev 1.16.5 or never.

RewriteEngine On

RewriteRule ^(?:fileadmin/|typo3conf/|typo3temp/|uploads/) - [L]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-l
RewriteCond %{REQUEST_URI} ^/typo3/.*$
RewriteRule ^typo3/(.*)$ %{ENV:CWD}typo3/index.php [QSA,L]

Hacktoberfest - Random Issue Picker

Tomas Norre Mikkelsen — Tue, 29 Sep 2020 20:26:24 +0000

The Hacktoberfest2020 is just around the corner, but where to contribute?

Andreas Frömer has created a small tool that can help you. The "Random Issue Picker" selects a random issue from GitHub based on certain criteria.

Requirements for the tool:

PHP 7.3+
composer

How to use it

$ git clone https://github.com/icanhazstring/random-issue-picker.git
$ cd random-issue-picker
$ composer install
$ bin/rip random:issue -l js --label hacktoberfest

This will then post information on the issue

Hamburger icon in dark mode
---------------------------

https://github.com/Ninja-Developers/Tech_Phantoms_website/issues/66


Hamburger icon in dark mode blends with the background. Icon color can be changed to white.

Now you can copy/paste or click the link to the issue and start working and earn your hackathon T-shirt.

Have fun, and have a great Hacktoberfest.

Codeception tests in Travis CI with DDEV and Selenium

Tomas Norre Mikkelsen — Sat, 02 May 2020 15:29:24 +0000

Recently I posted about how to run codeception with DDEV.
https://dev.to/tomasnorre/codeception-ddev-selenium-docker-36kk

This is of course really nice, but having it as part of the CI pipeline is even better.

So I decided to try to make it run within Travis CI which is my preferred public CI at the moment.

What to do?

Signup with travis-ci and registre your repository
Add a .travis.yml to your github repository
Make changes and push, and wait for the results of the tests.

You can see the step in the Travis CI Tutorial

The advantages here is that you will have your behaviour driven tests executed on every commit and pull request, to make sure you or others don't break functionality.

Configuring Travis CI

The .travis.yml is the configuration file for Travis CI. I contains different step and instructions to enable travis to do what you want.

This can be used to run PHPUnit, Checking Coding Standard, Codeception tests and so on, only your imagination is the limit.

This is how the .travis.yml looks like for my https://github.com/tomasnorre/ddev-codeception-poc

language: php
dist: bionic

notifications:
  email:
    on_success: never
    on_failure: never

before_install:
  # turn off XDebug for speed up
  - phpenv config-rm xdebug.ini || return 0

php:
  - '7.4'

services:
  - docker

cache:
  directories:
    - $HOME/.composer/cache

before_script:
  - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
  - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
  - sudo apt update
  - sudo apt install docker-ce
  - sudo curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  - sudo chmod +x /usr/local/bin/docker-compose
  - docker-compose --version
  - wget https://github.com/drud/ddev/releases/download/v1.13.1/ddev_linux.v1.13.1.tar.gz
  - tar vxzf ddev_linux.v1.13.1.tar.gz
  - sudo chmod +x ddev
  - sudo chmod +x mkcert
  - ./mkcert -install

jobs:
  fast_finish: true
  include:
    -
      stage: Tests
      name: "Code ception"
      script:
        - composer install
        - ./ddev --version
        - ./ddev start
        - php vendor/bin/codecept run

I had to install a never version of docker, manually to meet the requirements from ddev.

The result

Have fun.

TYPO3 Crawler with TYPO3 9 & 10 Support

Tomas Norre Mikkelsen — Tue, 14 Apr 2020 14:47:05 +0000

11th of November 2018, I added the typo3/cms-core:^9.5 to the composer dependency for the TYPO3 Crawler extension, the goal was set to make the Crawler compatible with TYPO3 9 LTS.

Unfortunately, it's been a very look journey, also way too long, as we are facing the TYPO3 10 LTS release in just a couple of days.

Finally, can I not only present a Crawler 9.0.0 which is compatible with TYPO3 9 LTS but at least as important also compatible with TYPO3 10LTS.

What has been added, changed or removed?

It's too much to list here so I'll recommend to browser through the changelog on Github.

Breaking Changes

Support for TYPO3 8
Support for TYPO3 7
PHP support for <7.2 @TomasVotruba
Remove root_template_pid option @bmack
Dependency for helhum/typo3-console
RealURL support
"legacy" Scheduler Tasks
nc_staticfile_cache hook not needed for TYPO3 9 LTS+

As integration the thing you need to pay most attention to is the Crawler Tasks, they need to be reconfigured as the Tasks classes are removed.

You will need to reconfigure them using the Execute console commands instead.

Thanks to ...

All these changes wouldn't be possible without help from others, or perhaps, but then we would have a release before TYPO3 11 LTS ;)

There have been 13 different contributors to the typo3v9 development branch, since the beginning. Thanks a lot to everyone pushing changes, fixes etc.

But of course also thanks to everyone contributing in the form of issues/bug reports and testing, this is also really important contributions, please keep doing that.

There are four persons, especially that I want to thank for helping out during this period of time. Three of them and I really got kick-started at the TYPO3 Developer Days 2019 (hereafter T3DD19), and I got the motivation back to really get it off the table.

You might know them already:

Benni Mack
Github Twitter

Benni has helped the Crawler a lot getting closer to the TYPO3 Core and using more standard PHP Libraries, to have less Crawler specific code. He is also always a good sparing when having questions on best practises.

He has for instance introduced/changed following into the crawler

Execution Strategy classes
Migrated FE hooks to PSR-15 middleware
Backend Module switched to Fluid templates
Implemented Site Handling

Tizian Schmidlin
Github Twitter

Tizian started working on the Crawler even before I got to know him in person, at the T3DD19. Tizian has been taking care of most of the Doctrine migration and implemented the new Workers that is also included in the TYPO3 Core to decouple the Crawler dependency in indexed_search.

Tobias Stahn
Github Twitter

I also got to know Tobias at the T3DD19, he had some question regarding the Crawler and we connected very well.

Since then Tobias has been very helpful with e.g implementing Extension API to handle extension settings. Besides that Tobias also has been helping with improving the documentation and done a lot of testing and review.

The last but not least person I want to say thank to is:

Tomáš Votruba
Github Twitter

Tomas isn't from the TYPO3 Community, but from Rector, he helped set up a better CI Pipeline with PHPStan, Rector, Coding Standards etc. This is something that will really benefit the TYPO3 Crawling in the future as it will be used in every commit forward in time.

Besides helping with the CI pipeline, he also helped clean up the code and removed support for PHP <7.2 which makes the crawler or more modern codebase. There are still work to be done, but with the help from Tomas and Rector, the next steps to deprecated PHP 7.2 will be easier.

Thanks again

I cannot pay them for their help in any way besides saying thanks as often as possible, if you want to tip them a little something, I think you should be able to find a Paypal, Github sponsoring, amazon wishlists or similar somewhere on their social profiles.

Some fun facts

As already mentioned I started this work on the 11th of November 2018, roughly one and a half year ago.

Since then we had:

~ 170 Commits
13 Contributors
319 files changed, 8236 insertions(+), 22502 deletions(-)

Bug and issues

If you find any bugs or issues in general with the crawler please create and issue on Github

Future ?

The next period of time, I will of course take care of the bugs and issues being reported.

My main work will be focussed on:

Fixing the Functional tests for TYPO3 ^10.3 - haven't proven any problems with the functionality, so decided to launch with 5 failing tests.
Adding behaviour driven tests, there are already a few added to the pipeline but want more, to ease the development further.
Add better test coverage in general
Refactor some of the "too big"-functions that make the maintenance to hard.

Currently, there are no big features in the pipeline, but hoping to continue improving the code base, so the next update will get smaller and closer to the LTS releases of the TYPO3 CMS.

I am looking forward to what the next time will bring,

#HappyCrawling!

How rector made my life easier

Tomas Norre Mikkelsen — Wed, 01 Apr 2020 05:15:45 +0000

In November 2019 I stared (*) the https://github.com/rectorphp/rector, as I just learned about how cool Rector was and how it could help me improve and migrate my legacy code.

My star on github, was as first used as a bookmark, but some days later, the team from Rector contacted me, and this change a lot regarding my view on refactoring, complexity and automation tools for development.

// Detect dark theme var iframe = document.getElementById('tweet-1196711403333332992-386'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1196711403333332992&theme=dark" }

What did I win?

I contacted the rector team as instructed in the tweet, and Tomas Votruba got back to me, and we discussed how he could help me.

As I'm most active with the TYPO3 Crawler extension I ask for help to migrate/optimise the current typo3v9-branch, as this is my work in progress branch for the next major release.

Tomas implemented following in our CI pipeline:

All this done within a week, and the code base was already improve a lot.

That's all?

The win wasn't "only" the 4 commits from Tomas, that helped me one time. He help me setup some basics that will help me a lot in the future.

CI

Now we have Rector CI, Code Standard and PHPStan checks as part of our Travis CI Pipeline and this is done for every single commit and pull request, so the quality of the code commit since then will also be improved, cleaner and easier to maintain in the future.

Development

Next time I want to deprecate a PHP Version e.g. PHP 7.2 and PHP 7.3, I can add PHP Sets to the ecs.yaml and the run the command .Build/bin/ecs check Classes Configuration Tests --fix and then most, if not all of the changes are done automatically within a few seconds.

The future for me

I don't see how my work as PHP Developer will continue without using a tool like rector, it makes a lot of things way easier.

The rector is in general a tool that can help a lot with PHP and framework updates like symfony, but as I'm more a TYPO3 kind of person, I've joined an initiative to make migration sets for TYPO3 Core Updates as well.

This will ease the upgrade from e.g. TYPO3 9 LTS to TYPO3 10 (Soon to be LTS), but this is something I'll share more about later.

Curl Proxy vs X-Forward-For

Tomas Norre Mikkelsen — Sat, 28 Mar 2020 14:13:12 +0000

When working with cUrl and proxies in PHP and a lot of request, first of all it can of course help to run multiple process at the same time, but besides that how you use the proxy itself can be a huge performance boost if done like below:

        $curlResource = curl_init($this->sourceUrl);
+       if (isset($this->getProxy())) {
+           $headerData[] = 'X-Forwarded-For: ' . $this->getProxy();
+       }
+       curl_setopt($curlResource, CURLOPT_SSL_VERIFYHOST, false);
+       curl_setopt($curlResource, CURLOPT_SSL_VERIFYPEER, false);
        curl_setopt($curlResource, CURLOPT_URL, $this->sourceUrl);
        curl_setopt($curlResource, CURLOPT_HEADER, 1);
        curl_setopt($curlResource, CURLOPT_FOLLOWLOCATION, 1);
        curl_setopt($curlResource, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($curlResource, CURLOPT_HTTPHEADER, $headerData);
-       // Set proxy if applicable
-       if (NULL !== $proxyIp && FALSE !== $proxyIp) {
-           curl_setopt($curlResource, CURLOPT_PROXY, $proxyIp);
-           curl_setopt($curlResource, CURLOPT_HTTPPROXYTUNNEL, 1);
-       }
```

`

This is a small diff from an example. The most important part is that instead of using the `CURLOPT_PROXY` and `CURLOPT_HTTPPROXYTUNNEL` then  use the `X-Forward-For` header do the trick for you.

The win here is due to they will be no network latency what so ever, as the proxy is only simulated and not actually used.

I'm using this test for GEO IP redirects etc. and when testing e.g. 300 redirects, then it does matter that you gain a four times faster job than you had before. With this said this of course depends on the proxy speed, but i'm close to guarantee that this will speed up you job, if you do this change.

Codeception Tests with DDEV and Selenium

Tomas Norre Mikkelsen — Sat, 28 Mar 2020 12:49:59 +0000

Running behaviour driven tests like Codeception can bring some advantages to your project, and will help you detect if you break functionality on updates, changes etc.

This blog post will help you setup the basics needed for developing your own tests and running them locally with ddev.

Prerequisits

DDEV installed on you local machine
Project setup locally with ddev, in this case I will use a plain phpinfo() site to lower complexity.

If you don't have a setup locally already, you can do this by follwoing the ddev get started guide, you can also close my repository on github, then you will not need to do the changes your self to check the prove of concept.

$ git clone https://github.com/tomasnorre/ddev-codeception-poc.git
$ cd ddev-codeception-poc
$ composer install
$ ddev start

If you choose the later one, you can skip the instructions in step 1-3, and go directly to executing the tests, but I would recommend you to do the steps manually.

1. Setup Selenium in DDEV

As you might already know ddev is based on docker, so to setup selenium, we need to add a custom docker-composer.selenium.yml to your ddev setup.

$ cd /path/to/your/local/project/.ddev
$ touch docker-compose.selenium.yaml

Use you default editor to edit the docker-compose.selenium.yaml file. This configuration is take from the Selenium Github Page, with a few adjustments, like the version: '3.6', container_name and removing the firefox and opera part.

version: '3.6'
services:
  selenium-hub:
    image: selenium/hub:3.141.59-20200326
    container_name: ddev-${DDEV_SITENAME}-selenium-hub
    ports:
      - "4444:4444"

  chrome:
    image: selenium/node-chrome:3.141.59-20200326
    container_name: ddev-${DDEV_SITENAME}-chrome
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    environment:
      - HUB_HOST=selenium-hub
      - HUB_PORT=4444

Now you can restart you ddev project with ddev restart and you will have a selenium container running together with your project.

2. Install Codeception

Steps copied from Codeception QuickStart Guide

$ composer require "codeception/codeception" --dev
$ php vendor/bin/codecept init acceptance

Let's prepare Codeception for acceptance testing
Create your tests and run them in real browser

? Where tests will be stored? (tests)
? Select a browser for testing (chrome)
  [0] chrome
  [1] firefox
 > 0
 Ensure that you have Selenium Server and ChromeDriver installed before running tests
? Start url for tests (http://localhost)
 Created test directories inside at tests
 Adding codeception/module-webdriver for WebDriver to composer.json
1 new packages added to require-dev
? composer.json updated. Do you want to run "composer update"? (y/n) y

After this we know have a codeception.yml in our root folder, next to composer.json and we have a tests folder, also in our root folder, that is created and populated with Codeception setup.

As I didn't adjust the start url as part of the initiation I need to adjust it manually.
Open the codeception.yml and adjust

enabled:
    - WebDriver:
        url: http://localhost/

enabled:
    - WebDriver:
        url: http://web/

this will make the Codeception make requests against the docker-container named web, which is your application container, in a default ddev setup.

3. Small Test example

In the tests folder we also have a file LoginCest.php which we are going to adjust, to add new tests.

The naming of the file, isn't important at this stage, but I would suggest to have files and structures match your application and what you're testing, this keeps it easier to find, correct, and add new tests.

Open the tests/LoginCest.php with your editor. Currently it looks like this

<?php
class LoginCest 
{    
    public function _before(AcceptanceTester $I)
    {
        $I->amOnPage('/');
    }

    public function loginSuccessfully(AcceptanceTester $I)
    {
        // write a positive login test 
    }

    public function loginWithInvalidPassword(AcceptanceTester $I)
    {
        // write a negative login test
    }       
}

The _before() is actions that will be executed before every test. The two other tests are empty right now so if you run php vendor/bin/codecept run you will have a green test result.

Let us add some more meaning full tests. First remove the two empty tests and add

public function canSeePHPVersion(AcceptanceTester $I)
{
    $I->canSee('PHP Version 7.4.3');
}

This will check if it finds the string "PHP Version 7.4.3" on the page / as that's the page defined in the _before() function.

Lets run the tests:

$ php vendor/bin/codecept run

Codeception PHP Testing Framework v4.1.4
Powered by PHPUnit 8.5.2 by Sebastian Bergmann and contributors.
Running with seed:


Acceptance Tests (1) -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
✔ LoginCest: Can see php version (1.78s)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Time: 2.91 seconds, Memory: 12.00 MB

OK (1 test, 1 assertion)

This page used can also be changed by doing

public function canSeePHPVersion(AcceptanceTester $I)
{
    $I->amOnPage('/index.php');
    $I->canSee('PHP Version 7.4.3');
}

If you have failing tests, there will be a link echoed in your terminal

There was 1 failure:

---------
1) LoginCest: Can see php version
 Test  tests/LoginCest.php:canSeePHPVersion
 Step  Can see "PHP Version 7.4.2"
 Fail  Failed asserting that  on page /index.php
--> PHP Version 7.4.3
System Linux ddev-codeception-poc-web 4.19.76-linuxkit #1 SMP Thu Oct 17 19:31:58 UTC 2019 x86_64
Build Date Feb 24 2020 18:39:14
Server API FPM/FastCGI
Virtual Directory Support disabled
Configuration File (php.ini) Path /etc/php/7.4/fpm
Loaded Configuration File /etc/php/7.4/fp
[Content too long to display. See complete response in '/path/to/Projects/ddev-codeception-poc/tests/_output/' directory]
--> contains "PHP Version 7.4.2".

Scenario Steps:

 3. $I->canSee("PHP Version 7.4.2") at tests/LoginCest.php:12
 2. $I->amOnPage("/index.php") at tests/LoginCest.php:11
 1. $I->amOnPage("/") at tests/LoginCest.php:6

Artifacts:

Png: /path/to/Projects/ddev-codeception-poc/tests/_output/LoginCest.canSeePHPVersion.fail.png
Html: /path/to/Projects/ddev-codeception-poc/tests/_output/LoginCest.canSeePHPVersion.fail.html

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

The png version is a screenshot, so if the text/element you are looking for are below the fold, you will not see it.

The html version is a scrollable version of the page, be aware that this might look quite rubbish, if the Javascript, css etc, isn't loaded correct, which often is the case, so might be difficult to spot the problem.

Conclusion

This is as you see a very small and compact example on how you can run Codeception in DDEV setup.

I'm far from expert on the field, neither ddev nor Codeception, but if you have any questions feel free to reach out to me on https://twitter.com/tomasnorre.

I have also pushed the example setup to my github account, so if you want to discuss something, that could benefit others, you can create an issue at https://github.com/tomasnorre/ddev-codeception-poc/issues