Karate | Test Automation Made Simple

Cosmin_Popescu · februarie 3, 2023, 8:45am

Mi s-a cerut code review si am dat de el. Este simpatic

Asa arata un test

gist.github.com

https://gist.github.com/ptrthomas/d5a2d9e15d0b07e4f1b46f692a599f93

hello-world.feature

Feature: karate 'hello world' example
Scenario: create and retrieve a cat

Given url 'http://myhost.com/v1/cats'
And request { name: 'Billie' }
When method post
Then status 201
And match response == { id: '#notnull', name: 'Billie' }

Given path response.id

This file has been truncated. show original

Features

Java knowledge is not required and even non-programmers can write tests
Scripts are plain-text, require no compilation step or IDE, and teams can collaborate using Git / standard SCM
Based on the popular Cucumber / Gherkin standard - with IDE support and syntax-coloring options
Elegant DSL syntax ‘natively’ supports JSON and XML - including JsonPath and XPath expressions
Eliminate the need for ‘Java Beans’ or ‘helper code’ to represent payloads and HTTP end-points, and dramatically reduce the lines of code needed for a test
Ideal for testing the highly dynamic responses from GraphQL API-s because of Karate’s built-in text-manipulation and JsonPath capabilities
Tests are super-readable - as scenario data can be expressed in-line, in human-friendly JSON, XML, Cucumber Scenario Outline tables, or a payload builder approach unique to Karate
Express expected results as readable, well-formed JSON or XML, and assert in a single step that the entire response payload (no matter how complex or deeply nested) - is as expected
Comprehensive assertion capabilities - and failures clearly report which data element (and path) is not as expected, for easy troubleshooting of even large payloads
Fully featured debugger that can step backwards and even re-play a step while editing it - a huge time-saver
Simpler and more powerful alternative to JSON-schema for validating payload structure and format - that even supports cross-field / domain validation logic
Scripts can call other scripts - which means that you can easily re-use and maintain authentication and ‘set up’ flows efficiently, across multiple tests
Embedded JavaScript engine that allows you to build a library of re-usable functions that suit your specific environment or organization
Re-use of payload-data and user-defined functions across tests is so easy - that it becomes a natural habit for the test-developer
Built-in support for switching configuration across different environments (e.g. dev, QA, pre-prod)
Support for data-driven tests and being able to tag or group tests is built-in, no need to rely on an external framework
Native support for reading YAML and even CSV files - and you can use them for data-driven tests
Standard Java / Maven project structure, and seamless integration into CI / CD pipelines - and support for JUnit 5
Option to use as a light-weight stand-alone executable - convenient for teams not comfortable with Java
Multi-threaded parallel execution, which is a huge time-saver, especially for integration and end-to-end tests
Built-in test-reports compatible with Cucumber so that you have the option of using third-party (open-source) maven-plugins for even better-looking reports
Reports include HTTP request and response logs in-line, which makes troubleshooting and debugging easier
Easily invoke JDK classes, Java libraries, or re-use custom Java code if needed, for ultimate extensibility
Simple plug-in system for authentication and HTTP header management that will handle any complex, real-world scenario
Cross-browser Web UI automation so that you can test all layers of your application with the same framework
Visual Validation via the built-in image comparison capabilities
Cross platform Desktop Automation that can be mixed into Web Automation flows if needed
Option to invoke via a Java API, which means that you can easily mix Karate into Java projects or legacy UI-automation suites
Save significant effort by re-using Karate test-suites as Gatling performance tests that deeply assert that server responses are accurate under load
Gatling integration can hook into any custom Java code - which means that you can perf-test even non-HTTP protocols such as gRPC
Built-in distributed-testing capability that works for API, UI and even load-testing - without needing any complex “grid” infrastructure
API mocks or test-doubles that even maintain CRUD ‘state’ across multiple calls - enabling TDD for micro-services and Consumer Driven Contracts
Async support that allows you to seamlessly integrate the handling of custom events or listening to message-queues
Mock HTTP Servlet that enables you to test any controller servlet such as Spring Boot / MVC or Jersey / JAX-RS - without having to boot an app-server, and you can use your HTTP integration tests un-changed
Built-in HTML templating so that you can extend your test-reports into readable specifications
Comprehensive support for different flavors of HTTP calls:
- SOAP / XML requests
- HTTPS / SSL - without needing certificates, key-stores or trust-stores
- HTTP proxy server support
- URL-encoded HTML-form data
- Multi-part file-upload - including multipart/mixed and multipart/related
- Browser-like cookie handling
- Full control over HTTP headers, path and query parameters
- Re-try until condition
- Websocket support

isti37 · februarie 3, 2023, 9:05am

Nu vrei sa stii cat de urat e sa te ocupi de Cucumber/Gherkin ca si programator.

Karate pare mai bun ca si JUnit cu Cucumber/Gherkin fiindca nu mai trebuie sa scrii codul pentru pasi separat de test. M-am luptat foarte mult cu code navigation-ul mizerabil la gherkin in IntelliJ.

La un proiect mic e posibil sa fie ok, dar cand ai 5 micro-proiecte intr-un monorepo cu aceeasi pasi (IDE-ul cauta dupa string) trebuie sa selectezi mereu la ce fisier vrei sa te duci. Asta daca le recunoaste.

In general fa orice poti sa nu folosesti Gherkin/Cucumber, iti faci de munca degeaba.

Partea buna in schimb, daca rulezi testele in CI/CD e ca pasul la care cade e mult mai usor de identificat si inteles. Problema e ca nu e sustenabil sa rulezi testele e2e in CI/CD, mai bine implementezi contract testing si unit teste care sa testeze functionalitate in loc de implementare.

tony · februarie 3, 2023, 2:30pm

Ok … poti sa detaliezi ?

isti37 · februarie 3, 2023, 4:01pm

In cazul Gherkin ai urmatorii termeni:
Given, When, Then, And, But

Ca termenii functioneze cu proprozitiile cucumber mai specifice tu o sa iti definesti un step definition file. In step definition o sa definesti string-ul care o sa defineasca ce sa faca acesti pasi. Aici scrii codul de automatizare, e.g. sa apese pe butonul cu rolul de ‘submit’ sau cu data-test-id-ul de ‘feature-x-modal-close’.
Deci prima data scrii testul cu pasii, dar dupa trebuie sa implementezi selectorii, in general vei avea o colectie de selectori/proiect. Daca ai putin noroc selectorii sunt direct exportati din front-end/aplicatie.
Daca te uiti atent in Karate ai doar pasi simpli, cum ai scrie ceva mai complicat vei apela la step definitions.

Cand ajungi sa ai 800-1500 de teste, o sa ai cateva sute de selectori custom.

Ce probleme apar:

O sa ai selectori dublati (fac acelasi lucru doar ca sunt scrise diferit). (prin selectori ma refer si la step definition dublat)
O sa ai selectori/pasi dublati intre mai multe proiecte daca ai monorepo, care pot sa faca ceva usor diferit, dar doar configuratia proiectului/IDE-ului decide din ce proiect vor fi luati step definitions. Pe VSCode nici n-am reusit sa le setez cand am avut mai multe proiecte, trebuia sa le rulez din terminal. Daca vrei sa rulezi un anumit test trebuie sa faci regex pe feature.
IDE-ul nu are de unde stii ca tu ai selectori dublati, nu o sa te atentioneze sa folosesti 'When I click the form submit button" in loc de “When I submit the form”. Fiecare om o sa isi implementeze proprii pasi (step definitions) cum nu reuseste sa scrie un test cu cei existenti sau nu functioneaza. Mai rau, va incerca sa adapteze logica la un step definition facut pentru ceva simplu sa mearga in general si posibil introduce flakyness la teste deja scrise.
QA posibil nu sunt programatori si tu va trebui sa implementezi pasii ceruti de ei oricum.
Te simti ca te-ai intors cu 10 ani in timp, flow-ul meu a fost sa selectez un pas, shift shift, incercam sa imi dau seama in care proiect e de fapt step definition-ul la testul la care ma uit.
Ca sa optimizezi step definition-urile oricum o sa iti faci helperi in framework-ul de automatizare (puppeteer/selenium/robot…), mai bine zis scrii o fateta pentru fiecare functie pe care o folosesti oricum, o sa ai 2-3 nivele de abstractie.
Pasii care sunt cu variabile/tabele sunt greu de cautat, poti avea un pas care are in comun cuvinte cheie.
Mult noroc la ChatGPT si automatizare de teste cu AI
E greu de mock-uit, nu vezi ce mock data folosesti sau formatul e acel tabel ciudat, trebuie sa te duci in 2-3 fisiere pentru preconditie.

Care e alternativa:
Aceeasi colectie de step definitions le faci ca functii/clase normale pe care iti merge autocomplete. Pui ce ai scrie in gherkin in comentarii si dupa vezi ce iti genereaza Github Copilot. Utilizezi functii in acelasi mod cu AAA, in Kotlin de exemplu poti scrie numele functiilor la fel cum le-ai scrie in step definitions cu ``. BDD se poate face si fara Gherkin/Cucumber.