FAQ on UMTS Testing
By Zahid Ghadialy (zahidtg@yahoo.com)
There are many kinds of testing
that needs to be performed on handsets. The most basic is the Conformance
test that is specified in 3GPP specification 34.123. Except that most handset
manafacturers perform more tests using the Network Simulators. Then they have
to perform IOT (Inter Operability test), Live air testing, Field Testing and
KPI (Key Performance Index) testing, etc.
There are many different types of
Conformance tests that the handset should be able to pass. Generally the
handset has to pass 95% of the tests to be GCF certified. The following are
the main categories of Conformance:
·
RF
conformance - This is based on 3GPP specifications 34.121. Recently due to
its size, it has been split into 34.121-1 and 34.121-2.
·
Protocol
conformance - This is based on 3GPP specifications 34.123. It is split into
34.123-1, 34.123-2 and 34.123-3.
·
USIM
conformance - This is based on 3GPP specifications 31.122.
·
Acoustic
conformance
Anyway GCF Certification is only
done for Protocol Conformance tests based on 34.123. These tests are written
in TTCN and are contained in 3GPP TS 34.123-3 drafted by ETSI on behalf of
3GPP. They are debugged and verified by individual SS vendors on their
platforms prior to submission for 3GPP TSG-T1 approval. Once approved by 3GPP
test cases are validated by an independent test house and submitted for
approval by the GCF. The GCF Conformance Test Toolkits provide the approved
test cases and validated protocol test systems required by UE manufacturers
and test houses for independent validation and GCF certification of a 3G UE.
To learn more about the GCF
certification see:
http://www.globalcertificationforum.org/static/?page=GCF_overview.htm
No its not. If the handset passes
Conformance Test, it can get GCF (Global Certification Forum) certification.
That proves that the UMTS handset is compliant to 3GPP specifications but is
not sufficient basis for the release of handset. All operators go through
their own rigorous testing before the handset can be released. Also there is
a requirement by all operators that the handset platform should have
undergone atleast two IOT's.
Is it possible to release a handset which is
not GCF certified?
Yes its possible. Finally it all
depends on the operator and at present (June 06) most of the UMTS handsets
are not GCF certified.
IOT's are very different to
conformance testing. Some of the UE's that pass all conformance tests do
perform badly in IOT's. Some of the differences are as follows:
·
In
Conformance tests, the number of neighbour cells are very limited. It is
nowhere like a real network. On the other hand in IOT's the number of cells
in Sib 11/12 are much more (sometimes upto 32). This means that there are
much more SIB segments as compared to conformance. UE's can have problem
handling these even though they pass Conformance with no problems.
·
In
Conformance tests the Cells are switched off and on immediately, whereas in
IOT's, their power is changed gradually, this can sometimes expose some flaws
in the UE's.
·
In
Conformance, generally the data path is not checked. For example a Voice call
is started and certain scenarios are simulated (like Handover, Relocation,
etc) the voice quality is not checked. We have come across UE's that do pass
the complete scenarios but instead of the voice call, all you hear is the
Ciphering noise. We have also seen some UE's where the voice path gets
dropped after 5 minutes in the call. All these cannot be tested in
Conformance tests.
·
Similarly
on the packet side you can check the Ping/Ftp is being properly performed in
IOT's that cannot be seen in Conformance tests.
No there are no written specs. All
networks do have their own set of tests that they need to perform for IOT's
but they are different for each network operator. The most basic set of tests
that need to be performed for IOT's are:
·
Voice
calls
o
To Test:
§ Call setup success
§ Call drops
§ Inter System Hand Off (ISHO)
§ BLER
·
Video
Calls
o
To Test:
§ Call Setup Success
§ Call Drops
§ BLER
·
PS calls
o
To Test:
§ PDP Context Activation Success
§ PDP Context Drop Call Ratio (DCR)
§ Throughput DL and UL
§ Ping Round Trip Time (RTT)
How to find out which Network Operators
perform IOT's? Are all the IOT's the same?
All Network operators have to
perform IOT's to prove that their UMTS network works with different UE's.
They are not all the same. Some of them are really tough. We have found that
Ericsson and Nortel are the toughest.
The IOT's are still controlled
environment and there is no check for RF, etc. In Field testing the UE's are
exposed to many networks simultaneously and the UE has to use the air
interface that can make lot of difference. Field testing also involves
testing for how the UE behaves in mobility environment and how good it is in
performing handovers, etc.
We know of a UE that would crash
when going through a particular street. After lots of investigations it was
found that the SIB segmentation in one particular cell was different and the
UE could not handle it properly, hence the crash. There are many many other
problems that only get exposed in Field testing.
The first thing you should know is
what particular scenario you are interested in. Once you know that then you
should get the latest version of 34.123-1 from the 3GPP server and try to find a test case
that will match your scenario. Once you have found that then you can find the
UE log on the 3GPP server for that particular scenario. Unfortunately you will
have to go through all the directories to find the test case. There is no
shortcut i am aware of. Some of the old tests are not in TSG-RAN but in TSG-T.
|