Health & Lifestyle

Regression environment for Nx-builder tool suite IP integration tooling

Description
Eindhoven University of Technology MASTER Regression environment for Nx-builder tool suite IP integration tooling Saksena, M. Award date: 2007 Link to publication Disclaimer This document contains a student
Published
of 45
0
Published
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Similar Documents
Share
Transcript
Eindhoven University of Technology MASTER Regression environment for Nx-builder tool suite IP integration tooling Saksena, M. Award date: 2007 Link to publication Disclaimer This document contains a student thesis (bachelor's or master's), as authored by a student at Eindhoven University of Technology. Student theses are made available in the TU/e repository upon obtaining the required degree. The grade received is not published on the document as presented in the repository. The required complexity or quality of research of student theses may vary by program, and the required minimum study period may vary in duration. General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. Users may download and print one copy of any publication from the public portal for the purpose of private study or research. You may not further distribute the material or use it for any profit-making activity or commercial gain Masters Thesis Embedded Systems Regression environment for Nx-Builder tool suite IP integration tooling By Mudit Saksena Supervisors Prof Dr. Ir. Jef van Meerbergen (TU/e) Ir. Marino Strik (NXP Semiconductors) VER 6.0 2 Acknowledgement The author expresses his deep sense of gratitude to Ir. Marino Strik, System Architect, NXP Semiconductors and Prof Dr.Ir.Jef van Meerbergen, Electrical Department, Technical University, Eindhoven The Netherlands for their precious counsel and meaningful supervision of this work. He also thanks all those who rendered any direct or indirect help in completion of this work at the University, Philips and NXP. He is particularly indebted to his brother Mr. Abhishek Saksena who often rendered valuable advice during the course of this work. He is also sincerely thankful for consistent encouragement provided by his father Dr. K.K.Saksena, mother Mrs. Rekha Saksena, sister Ms. Anjali Saksena and the sister in law Mrs. Neha Saxena. Mudit Saksena Student MSc Embedded systems Technical University Eindhoven December 4 DEDICATED TO THE ELECTRONICS & SEMICONDUCTOR INDUSTRY ACROSS INDIA, NETHERLANDS AND THE GLOBE 5 6 Table of contents Contribution...9 Keywords...10 The terminology...12 Introduction...13 Regression start...18 A Nx-Builder Regression run...20 Regression testing...21 Problem statement...25 Challenge(s) and Proposal(s)...26 Implementations ) Nightly run...28 Approach...28 Analysis ) Coverage...29 Code coverage...31 Approach...33 Coverage results...35 Analysis...36 Technique for increasing code coverage...36 Functionality coverage...36 Approach...37 Coverage Results...40 Analysis...40 Conclusion and future work...46 Contribution s...46 Future work...47 References Table of figures FIGURE 1 - DESIGN HEAD COUNT...13 FIGURE 2 - NX BUILDER...15 FIGURE 3 - SCHEMAS...17 FIGURE 4 - REGRESSION PROTOTYPE...18 FIGURE 5 - A NX BUILDER REGRESSION RUN...20 FIGURE 6 - REGRESSION TESTING...21 FIGURE 7 - PARALLEL JUNIT...27 FIGURE 8 - LINE COVERAGE...35 FIGURE 9 - IMPROVED REGRESSION ENVIRONMENT...41 FIGURE 10 - JAVA COMPILATIONS...42 FIGURE 11 - OFFLINE INSTRUMENTATION...42 FIGURE 12 - CODE COVERAGE RESULTS GENERATION...43 FIGURE 13 FUNCTIONAL COVERAGE RUN...44 FIGURE 14 - FUNCTIONAL COVERAGE RESULTS GENERATION...44 FIGURE 15 - LOAD DISTRIBUTION Abstract It is always preferable to release software in a 'known' state compared to software in an unknown state, which may contain surprises for both the developers and the users! Making changes to software, which is in a known state can, and often does, pose a serious threat to that known state. Every developer knows that even the smallest change can render software useless if the implications of the change were not properly understood or if the change was insufficiently tested during and after implementation. Changes to software are generally unavoidable. Eventually the need arises to amend software, either to correct defects or to modify functionality, and those changes may be required at short notice. It is under these circumstances that the investment in automated regression testing tools can begin to pay back big time. Regression testing is the name given to the process of verifying the software after such modifications and to keep track of error history. The goal of the Nx-builder regression testing is to verify a Nx-Builder architecture template during its development cycle. Contribution Successful integration of automated regression testing tools into the development process offers greater levels of confidence that changes introduced into software at short notice will not go without testing nor will they cause unexpected consequences when the software is later shipped to users In this thesis I presented the EMMA java code coverage tool and a software code to serve as an integral part of the existing regression environment. As the main contribution I have shown how to integrate and highly automate the different parts of the new regression environment, which includes writing software code, scripts and integrating EMMA tool. I used the load-sharing facility to schedule the tests on Multicomputer system leading to a successful overnight regression run consisting of many designs and their steps, finally generating the necessary reports. I have also improved the coverage reports to industry standards. 9 Keywords IP IPYP HW SW HDK SDK IPK FE RT Nx-Builder VSW API CoReUse Platform RTG SoC HTML XML Baseline Configurator Generators Intellectual property IP yellow pages = IP Library Hardware Software Hardware Development Kit Configurable IP extraction tool Software Development Kit Interaction Development Kit Front end Regression testing Nexperia Platform Builder Verification software Application Programming Interface -a set of routines, protocols, and tools for building software applications. Reuse coding rules for hardware designs Platform deliverable that consists of IP blocks and generators that allow you to construct a fully functional SoC design (constrained by architectural rules) from building blocks + software commands. Reuse Technology Group System on Chip - Complex IC that integrates the major functional elements of a complete end product into a single chip. Hypertext Markup Language Extensible Markup Language A baseline is a specification or a product that has been formally reviewed and agreed upon which there after serves as the basis for future development. A baseline version is a system that has passed a test suite. Is a java routine that is run when a component is added to a design, or when a user decides to configure a component, bus or bus interface. Are hand written java routines which can be executed by the end-user through the GUI menus, these can be used for flow integration or for generation of components using HDK or IPYPCLI 10 JVM GUI CLI XML Schema PLX Java Process Metadata Coverage data Java Virtual Machine Graphical user interface Command line interface The purpose of an XML Schema is to define the legal building blocks of an XML document, just like a Document Type Definition. Platform express based design Object-oriented programming language An instance of an executable program A set of binary Java class descriptors that record details of class structure like the number of methods in a class, their name and basic block structure. As long as the original classes are not recompiled, the same metadata can be used for any number of coverage profiling runs A set of binary Java class descriptors that reflect a particular coverage profiling run (which method and basic blocks were hit for which class, etc). As long as they correspond to the same metadata, several such data sets can be merged together 11 Section 1 The terminology IP core - a reusable unit of logic, cell, or chip layout design and is also the property of one party -- hard cores, firm cores, or soft cores Design A description of a design in XML describing the various IP s and how they are interconnected Design Database A set of designs. Step an action such as Build, simulate and synthesis on a design. Combination - {Design: one or more actions}. Sequences An ordered set of combinations. Suite A collection of test sequences. Server farms -a collection of computer servers usually maintained by an enterprise to accomplish server needs far beyond the capability of one machine Log - recording which tests cases were run, who ran them, in what order, and whether each test passed or failed Out Log - contains detailed concatenated information from various tools such as build, simulation and synthesis and also what parts of the program / tool suite were executed and how many times Regression testing - type of software testing which seeks to uncover regression bugs. Regression bugs occur whenever software functionality that previously worked as desired stops working or no longer works in the same way that was previously planned. Typically regression bugs occur as an unintended consequence of program changes 12 Section 2 Introduction The electronics in today s consumer goods is converging to a small number of processor based platform architectures, which are becoming standard across the whole industry. This means that to continue to make money from this type of silicon the hardware now needs to be created as cheaply and quickly as possible. In the ReUse Technology Group of NXP Semiconductors an internal project known as Nx- Builder had been researching a development environment for enabling the integration of fixed and configurable System IP. In today s market there is an increase in the complexity of digital system-on-chips (SoC), which leads to an increase in resources to execute designs. 80 % of the costs are headcount related. * Figure 1 - Design head count (* Source Global system IC service management report, IBS INC. July 05) 13 To keep up with these challenges in the market there is a need to develop, verify and produce SoCs with more complexity in shorter development period. There are changes needed in the development model to drive improvements in effectiveness to reduce cost and improve time to market Thus a generic framework is a necessity which can enable platform developers to implement their systems in a consistent, flexible and easy-to-use manner resulting in the reduced SoC development cycles for improved time-to-market and automating the verification process for reduced cost of Non-Quality. This has already led to the creation of a Building Block Library called IPYP (IP Yellow Pages). This Library includes the basic components from which the majority of platforms are built. Each of these blocks has several configuration parameters that need to be tailored to a specific platform instance. To produce a platform instance these blocks have to be configured and then have to be assembled together to achieve the particular platform instance desired. This assembly then has to be verified to ensure correct construction and finally the platform passes through a series of implementation stages before reaching silicon. The building blocks include components, buses and connectors. The components are IP such as memories, peripherals etc. All components will have one or more bus interfaces. The buses are master interfaces. The connectors (basically IP s) are slave bus interfaces and may require connection to other components. Nx Builder Nx Builder is an integration of a selected number of tools (as shown in Figure 1 below). Its objectives are: To form a highly automated SoC design flow for hardware To enable the rapid development and deployment of platform-based sub-systems and onto full IC s (Figure 1 - Block B) To make a significant impact on the time to market for derivative designs and the deployment of a standard design practice. 14 Figure 2 - Nx Builder Nx Builder is currently based on Mentor Graphics Platform Express. The Nx-Builder development environment will ultimately support the 3 main phases in the development of SoCs: Integration & verification of IP (IP integration) Synthesis, DfT insertion and initial steps towards physical design (Chip design) Architecture exploration & early software development (ESL domain) Nx-Builder exploits the fact that most platform designs differ by relatively small amounts from each other and follow a small number of standard templates e.g.: Some peripheral blocks change Some block parameters change 15 Bus structure changes. By reusing results for unchanging portions of the design, users can save many hours of processing or manual intervention during iterations, greatly reducing the time-to- market for large and high-speed designs. Nx-Builder uses this incremental methodology to allow the designer to enter details of how a platform design differs from a standard template and utilizes this information to interface with the various CTO design flows. The Nx-Builder tool consists of the following components. The Users can build libraries on top of Nx-Builder. Platform Express Nx-Builder library 1. Nx-Builder API 2. Nx-Builder flows Nx-Builder is based on a third-party solution called Platform Express from Mentor Graphics. Platform Express provides as part of its core functionality a graphical user interface as well the basic routines required for integrating IP as well as creating and building designs. Nx-Builder allows a user to compose a sub-system using an XML view of IP, which is available in a repository. All information required to perform user interaction and context analysis is present in this XML catalogue view. When a user is ready with the input side, then a build of the system starts the process where Nx-Builder derives from user input and context analysis the configuration parameter values for each IP. Next IP is configured and downloaded from the IP repository using these configuration parameters. The downloaded IP provides a new more detailed XML description of the IP and it replaces the initial catalogue view XML. The detailed XML has all information required for the generation of a RTL top-level. Furthermore it provides all information for an automated simulation and implementation flow, which writes out scripts and control data for other EDA tools. Software architecture Figure 2 explains where Nx-builder API s and Generator s sit. On top of the Platform Express core sits a public Java-based API that allows extending the tool with new functionality. PX also provides an XML schema for encapsulation of IP through metadata written in XML. The Nx-Builder verification flow reuses large parts of the Platform Express verification flow. On top of the Platform Express XML schema sits the new standard SPIRIT schema. This means that Platform Express can read both their own proprietary XML format as well as the standard SPIRIT format. It is even possible to mix the two in the same file. The advantage of 16 that is that one can make use of new schema elements in the Platform Express schema before they become available in the standard SPIRIT schema. Nx-Builder is a new API with Philips-specific extensions as well as a number of generators that extend the PX verification flow and also provide new flows such as synthesis. An important part of the Nx-Builder API is a link to HDK and IP Yellow Pages IP repositories. On the metadata side, IP and IP generators can make use of the SPIRIT schema and the PX schema and on the generator side they can use both the PX as well as the Nx-Builder API. Platforms sit on top of the whole lot and can be delivered through platform templates as well as generators that implement design constraints etc Figure 3 - Schemas 17 Section 3 Regression start The Regression framework for Nx-Builder tool is built upon the JUnit framework and uses the Nx-Builder and Platform Express java API. JUnit is a simple, open source framework written in java to write and run repeatable tests. Mentor provides java API to instantiate Platform Express, load a design and execute generators in batch mode. The regression classes make use of the above-mentioned API along with the Nx-Builder API to provide a framework for tests. The functionality to be tested can be specified via XML and can be implemented as a java class. The following figure shows the architecture of the regression environment at present. Figure 4 - Regression prototype The regression framework consists of a number of regression designs, regression framework classes, GUI classes, and the actual steps(s) implemented in java. The regression framework classes consist of following important components - 1. Test Runner - Runs the test 18 2. Result Collector - A running test throws a lots of results, which are collected by result collector 3. Test Case Interpreter To interpret the information from the XML file(s). 4. Report Generator Generates the report(s) for analysis from the various results obtained 5. NxJunit Test This component helps to communicate with platform express. The deisgn.plx file is standard input xml file describing the various IP s and how they are interconnected The TestCase.xml file is a file that consist information about the test case, logins and all other information needed for a particular test. This file is generated when a user configures some test cases for his own regression run. These configurations can only be created with the GUI and can be made to run with GUI or CLI. The output of a regression run is login (HTML O/P) file, which gives the various pass / fails results of the tests. As of now the logins are generated after each finished test. The login provides some information about the steps if they have failures. A final summary report is generated after all the test logins are obtained. Many log files are generated which captures all the output information from the various tools during the regression run. The GUI can be used to configure and store steps into suites. The suites can be run directly from the GUI or from the CLI. The steps are available in two flavours - API steps Unit API tests for checking the structure of the PX Index. - Functionality steps To test the coverage. The unit tests are used to check the correctness of every public method inside the Nx builder javasrc folder. This unit tests also includes some environmental tests for e.g. Check generator test - Checks that the environment is set up correctly by accessing a method in the Nx-Builder class Check library test - A JUnit tests the case for regression testing, checks the environment is set up correctly by looking for one cell in a Mentor library and one cell in a Philips library. Import test - Tests the CoReUse functionality using a simple test design to check basic flow. The functionality test is used to test the coverage aspects of the Nx-Builder tool suite, for example: synthesis, front-end flow and simulation. The tests run in the order in which they are specified 19 Section 4 A Nx-Builder Regression run Figure 5 - A Nx Builder Regression Run As can be seen in the above figure a user selects design(s) from a design database, he then selects an action(s) to test per design If needed a time out can also be specified per design in seconds A user can also selects more than one design in an XML file. The XML file forms a sequence There can be many sequences for a regression run forming a suite Finally a suite is made to run on Multicomputer distributes system When an action is executed on a design during the regression run, it makes use of the Nx- Builder java code and the JUnit 20 Section 5 Regression testing Regression testing is a testing process that is applied after a program is modified. Once software has passed all its initial tests it is important to continue to test it regularly since environmental and other changes (modifications) can introduce new errors. Regression testing involves testing the modified program with some test cases in order t
Search
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks
SAVE OUR EARTH

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!

x