TestLink vs XStudio ?

Discuss test processes, standards, documentation, teams, criteria, test environments, etc.

Moderators: Amaradana, TurboPT, TL Developers

TestLink vs XStudio ?

Postby lsarandon » Sun Apr 26, 2009 5:35 pm

Hi
I'll be working soon for a customer who generally uses Testlink. Unfortunately, I'm used to work with XStudio (which is also a free test management solution: http://www.xqual.com) for a few time now.

So, I'm in the process of learning about Testlink. Please, excuse me if I have some dumb questions.

I just looked at the documentation and there are a few features that I like very much on XStudio that I did not find. Maybe it's not documented or available on a different form... Could you tell me if they are already existing or if there are some plan to deliver them in a short time?

This includes:
- project management (i.e. Scrum) integration (it allows me to check availability of the QA team members through calendars (to spread tasks such as development of automated tests and execution/analysis of the results)
- scheduling of the automated tests (I'm used to schedule my test campaigns to start and run automatically overnight on different PCs when they are time consuming or CPU intensive so that I get the results in the morning)
- coverage metrics from requirements to specs, specs to test or requirements to tests
- Control the progress in time on the development of my tests (I can produce some graph that helps me to estimate when the developments will finish but also shows progress to my customers)
- filtering tests based on attributes (this is then very easy to select a large number of tests without selecting them manually one by one)
- is it possible to interface TestLink with external proprietary tests (is there the equivalent notion of Launcher that I'm used to develop with XStudio)?
- For manual tests, XStudio displays a list of simple questions to the test operator. How does it work with TestLink? I did not really understand from the documentation.

On the other hand, I like the fact that testlink allows to generate OpenOffice or Word document. XStudio generates only HTML documentation :(

If you can help me with those simple questions it would be great so I can decide if I can switch to Testlink for this project.

Thank you,
LS.
lsarandon
TestLink user
 
Posts: 3
Joined: Sun Apr 26, 2009 4:39 pm

Postby havlatm » Sun Apr 26, 2009 9:30 pm

TestLink is matured tool with large number of features. I don't know xqual tool. So I cannot compare.
There is my notes for your topics:
- TestLink collaborates with bug trackers that cares about dev. tasks. So integration of task management is not priority.
- Night tests are run by feature testing tools (no need for TL). You should just deliver results via web-services to TestLink DB. That's how users do.
- REQ coverage is implemented (include reports).
- Metrics and graphs are available for progress control. In addition you can define milestones for testing progress (include test prioritization).
- Testers can define both 'keywords' and 'custom fields' for filtering purpose.
- Interface to external proprietary tests ... TestLink has own API to accept request from other tools. There is also experimental possibility to run these tests from TL GUI. I'm not familiar with. It generally allows to call an external tool API.
- My experience is that checklists are not wanted during test design. It's enough as a part of training. The next version will have test patterns, that could be used as a checklist.
havlatm
Member of TestLink Community
 
Posts: 940
Joined: Mon Oct 31, 2005 1:24 am
Location: Czech

Postby lsarandon » Sun Apr 26, 2009 10:17 pm

Thank you havlatm,

- TestLink collaborates with bug trackers that cares about dev. tasks. So integration of task management is not priority.

Well, integrating with bug-tracker is a necessity. XStudio and Testlink are doing so. But, I was speaking about management of resources during the time of the "project". Bug-trackers do not provide metrics such as "do I have enough resources between 1st of January and 28 February" to develop all my test scripts + validate the SUT XX version YYY. This is one of the things I appreciate in XStudio or in some of the commercial (and very expensive) products I worked with.


- Night tests are run by feature testing tools (no need for TL). You should just deliver results via web-services to TestLink DB. That's how users do.

Ah! so it means that scheduling and execution must be handled by the testing tools themselves? hum... for me the testing tools were supposed just to be able to execute scripts and return results. for example if I interface Testlink with AutoIt how can I manage scheduled execution?
I guess there are a lot of testing tools that just provide an execution/automation framework without any "management layer".

- REQ coverage is implemented (include reports).

Cool!

- Metrics and graphs are available for progress control. In addition you can define milestones for testing progress (include test prioritization).

Excellent! so Am I right in saying that I would be able to track implementation of test scripts + executions of test campaigns?

- Testers can define both 'keywords' and 'custom fields' for filtering purpose.

Good again. Are the custom fields strongly typed? for example, would I be able to filter "all tests that have attribute1<10 AND (attribute2.equals("foobar") OR attribute3==false)" ? XStudio uses a tree to define these kinds of boolean expression. How does it work with Testlink? I didn't see any screenshot on this aspect...

- Interface to external proprietary tests ... TestLink has own API to accept request from other tools. There is also experimental possibility to run these tests from TL GUI. I'm not familiar with. It generally allows to call an external tool API.

Ok, I'll try to have a look at the API on the website to see how I could interface with the test scripts I already wrote.

- My experience is that checklists are not wanted during test design. It's enough as a part of training. The next version will have test patterns, that could be used as a checklist.

hum.. don't get it. So what happens when a test operator run a test campaign then? how does the test operator tells Testlink that testcase X failed and testcase Y succeeded? how does he provide his comments? how are the traces logged in the DB?

Thanks for your useful answers, I understand better now even if I still have lots of questions as you can see... ;)

Thank you again,
LS.
lsarandon
TestLink user
 
Posts: 3
Joined: Sun Apr 26, 2009 4:39 pm

Postby havlatm » Mon Apr 27, 2009 7:55 am

IMO You mess tools. TestLink targets test management.
For people management there are different tools like is dotProject and more. I believe that xstudio cannot do job better than these tools. You can assign testing to people in TestLink and several other work assignment will be implemented in new versions. I personally have not ambition to improve these area. You can of course create own modification that exactly fits your needs.

The similar situation is regarding test automation. It's wasting of time to write own framework for test running if there is amount of ready to serve tools. This is one additional layer for the process. In any case you need some OS related trigger for Linux it's cron. All tools that offers such feature do it similar way. Own application service (or wrapper) or generic service is the only formal difference.

I do not understand your last question. I expected that you ask some checklist for test design phase. But your comments from previous replay are related to test execution. It looks that you asking about possibility to see test case and mark test result ... this is really basic behaviour. I suggest to browse our User manual at first. (Link 'Documentation' at the top of these page).
havlatm
Member of TestLink Community
 
Posts: 940
Joined: Mon Oct 31, 2005 1:24 am
Location: Czech

Postby lsarandon » Wed Apr 29, 2009 8:54 pm

IMO You mess tools. TestLink targets test management.
For people management there are different tools like is dotProject and more. I believe that xstudio cannot do job better than these tools. You can assign testing to people in TestLink and several other work assignment will be implemented in new versions. I personally have not ambition to improve these area. You can of course create own modification that exactly fits your needs.

I understand your point. However, what I can observe everyday is that:
1) it's quite rare to find companies that use tools like dotProject
2) if you want to follow this way, you'll have to multiply tools (probably tens of softwares... to make interoperable... not easy)
For sure XStudio is not the best for project management but it's simple and efficient. You could argue also that some other tools handle also better requirements and specification. But here again, the main pro is that everything is integrated.
Anyway, I 100% understand your point and it makes sense.


The similar situation is regarding test automation. It's wasting of time to write own framework for test running if there is amount of ready to serve tools. This is one additional layer for the process. In any case you need some OS related trigger for Linux it's cron. All tools that offers such feature do it similar way. Own application service (or wrapper) or generic service is the only formal difference.

hum... in this particular case, I do not really agree. Most of the automation tools I worked with in the past either:
1) did not include scheduling facilities
2) was including this facility coupled with very basic and ineffective test management systems.
3) if you 'externalize' the scheduling (cron or similar...), then
you must have an architecture where the test will connect to the test management framework. this just sounds weird to me. XStudio's approach - as some others - consists in having the test management framework to drive the tests and not the opposite way.

I do not understand your last question. I expected that you ask some checklist for test design phase. But your comments from previous replay are related to test execution. It looks that you asking about possibility to see test case and mark test result ... this is really basic behaviour. I suggest to browse our User manual at first. (Link 'Documentation' at the top of these page).

I read the doc and I now understand how testlink deals with manual tests.
This is really different from what I knew.


Thanks havlatm for your support. Very quick and effective.

However, for this particular project, I will probably continue with what I'm used to - I'll definitely keep an eye on TestLink in the future though.

Thanks again for the great support.
LS.
lsarandon
TestLink user
 
Posts: 3
Joined: Sun Apr 26, 2009 4:39 pm


Return to General Testing and Test management



Who is online

Users browsing this forum: No registered users and 1 guest