NetBSD on Marvell Armada XP System on a Chip - Semihalf · NetBSD on Marvell Armada XP System on a...

NetBSD on Marvell Armada XPSystem on a Chip

EuroBSDCon 2013, Malta

Zbigniew [email protected]

1

mailto:[email protected]

mailto:[email protected]

NetBSD on Marvell Armada XP System on a Chip

Presentation outline

‣ Justification for choice

‣ Hardware overview

‣ Porting to a modern ARM

‣ Prerequisites

‣ Essentials

‣ Device drivers

‣ Validation

‣ Integration to mainline NetBSD

‣ Comparison with the mainline FreeBSD

2


Justification for choice

‣Why should we be interested in ARM?

3



‣Why should we be interested in ARM?‣ Industry trend to save energy

‣ High performance solutions = new markets

4





‣Why we invested our work in ?

5





‣Why we invested our work in ?‣ Customer requirement

‣ Known of it’s portability

‣ Friendly license

‣ Active community

‣ Good alternative to ”other similar”

6



‣ Advantages‣ Building for ARMv7 supported

‣ Memory management supported

‣ Whole set of drivers for older Marvell chips

‣ Disadvantages‣ No settled support for ARMv7 platform (at the time)

‣ No SMP for ARM

7


Hardware overview

‣ Marvell Armada XP‣ Quad core PJ4Bv7

‣ 2MB shared L2 cache

‣ Hardware cache coherency, I/O coherency

‣ Integrated storage (USB 2.0, SATA II, NAND, NOR...)

‣ High speed networking (NETA)

‣ PCI-Express

‣ Cryptographic engines

‣ DMA engines

‣ Advanced power management

8


Porting to a modern ARM

6.0-RC1 / BEAGLE 6.0.1 / ARMADAXP

‣ Prerequisites‣ Arrange booting environment

‣ Define code base

‣ Compiler availability

‣ Support for similar platform

9



‣ Prerequisites‣ Understand work to be done

‣ Is the CPU core supported?

‣ Is the SoC family supported?

‣ Is the particular board supported?

‣ What peripheral devices are supported?

‣ Armada XP

➡ New CPU core - PJ4B➡ New SoC family - Armada XP

➡ New evaluation boards➡ Some peripherals matching existing drivers

10



‣ Prerequisites‣ Arrange configuration files

‣ Processor specific under sys/arch/arm/marvell/files.marvell

‣ Defines core device specification for whole chip family

‣ Evaluation board specific under sys/arch/evbarm/conf/

‣ Board specific files and option definitions → files.armadaxp

‣ Extra specification for Makefile → mk.armadaxp

‣ Set of board specific configuration options → std.armadaxp‣ Kernel configuration file → ARMADAXP

11



‣ Essentials‣ CPU core support

‣ What does it mean?

12




‣ What does it mean?

NetBSD/arm utilizes a set of ”cpu functions” defined in:

sys/arch/arm/include/cpufunc.h (see struct cpu_functions)

‣ Function wrappers for basic CPU-related operations such as:‣ cache manipulation routines

‣ core detection and configuration (including MMU)

‣ read/write buffers, branch predictor management

‣ other

13




‣ Do not increase redundancy - sufficient CPU functions can be reused

‣ Remember of the core bugs and/or erratum

Example:ENTRY(pj4b_setttb)! /* Cache synchronization is not required as this core has PIPT caches */! dsb#if defined(L2CACHE_ENABLE) && defined(AURORA_L2_PT_WALK)! orr! r2, r0, #TTB_FLAGS_UP#else! bic! r2, r0, #0x18#endif! mcr! p15, 0, r0, c2, c0, 0! /* load new TTB */! mov! r0, #0! isb! mcr! p15, 0, r0, c8, c7, 0! /* invalidate I+D TLBs */! RET

14




1. Create custom CPU functionssys/arch/arm/arm/cpufunc_asm_pj4b.S

2.Add new CPU to configuration filessys/arch/arm/conf/files.arm

sys/arch/evbarm/conf/ARMADAXP

3.Add CPU class and CPU ID to auto-detectionsys/arch/arm/arm32/cpu.c

4.Fill-up cpu_functions structure with the desired CPU ops

4.1.Implement <cpu>_setup routine

15



‣ Essentials‣ Start code

Assumptions:• Basic SoC initialization is done by firmware

• First part of the kernel to be executed

• Start code file and object is defined in mk.armadaxp • Written in Assembly language:

• To allow relocation• No environment for C code

Goals to achieve:• Remap kernel to the true VA

• Redirect to locore.S for stack initialization

16



0xC0000000

0x00000000

MMU ON

MMU OFF

VA PA

‣ Essentials‣ Start code

17



‣ Essentials‣ Machine initializationinitarm() from

sys/arch/evbarm/armadaxp/armadaxp_machdep.c• First C code executed

• Includes all actions that need to be performed before jumping to main()

Goals to achieve:• Bootstrap virtual memory!

• Create initial page tables• pmap_bootstrap()• Map devices for bus_space_map()• Allocate and create mappings for exception vectors page,

message buffer, stack, etc.• Inform UVM about the available pages

18






Goals to achieve:• Initialize all crucial chip features

• CPU (functions)• Decoding windows• Clocks (time base)• Interrupts (preconfigure)• L2 cache • I/O cache coherency

19






Goals to achieve:• Other important configurations

• Abort handlers• Stack pointer• System console• KGDB

20



‣ Essentials‣ L2 cache management

L1 CACHE

L2 CACHE

CPU

CACHE

21



‣ Essentials‣ L2 cache management

1. Set L2 policy (LRU, WB)

2. Enable cache and TLB maintenance broadcast

3. Invalidate cache

4. Set PoC and PoU

5. Finally... enable L2 cache

‣ CPU functions for cache maintenance will do the rest

22



‣ Essentials‣ I/O cache coherency

‣ The Coherency Fabric

• i.a. forwards transactions initiated by the I/O devicesand tagged with a shared attribute onto theprocessors snoop engine. On-chip cachescan then snoop these transactions

↓COHERENT DMA

23




‣ The Coherency Fabric

CPU

L1 CACHEL2 CACHE

DRAM controller

BUS DEV

DEV

COHERENCY FABRIC

24




‣ Adjustments (hacks) to the bus_dma and pmap

‣ Set the pages attributes to shared‣ Override default non-coherent settings in‣ _bus_dmamap_load_mbuf()‣ _bus_dmamap_load_buffer()

‣ Add HW synchronization barrier to ‣ _bus_dmamap_sync()

NO NEED FOR SW CACHE MAINTENANCE ON I/O!

25



‣ Essentials‣ Atomic operations

‣ Issues with atomic byte swapping required for lockingDeprecated swp and swpb instructions resulted in undefinedinstruction exceptions in kernel.

• Exchanged to ldrex strex pair• Added barriers after acquiring the lock & before releasing

‣ More issues with the building distribution.Undefined instructions (swp & swpb) in shared libraries.• Need for passing ARM_ARCH_7 definition to distribution build• Inserting barrier instruction opcode for non-kernel builds

DMB ! .word 0xf57ff05f

26



‣ Essentials‣ Interrupts

Multiprocessor Interrupt Controller

CPU

MPIC

DE

V0

DE

V1

DE

V2

DE

V3

1 2 4 2

4HW prioritization

27



‣ Essentials‣ Interrupts

Multiprocessor Interrupt Controller

• Great for implementing NetBSD SPL mechanism• Increasing/decreasing prio done in HW• Controller takes care of proper interrupts

notification• Much faster and simpler than the solution for older

Marvell chips

• All pic_ops needed to be implemented- establishing- blocking- unblocking

28



‣ Essentials‣ Marvell SoC abstract bus (mvsoc)

‣ Armada XP support attached to the existing mvsoc

‣ mvsoc bus uses direct configuration method‣ Resource description is not taken from the

array dynamically created by config(8) basingon the configuration nodes from kernelconfiguration file

‣ All resources are defined in advance‣ Bus needs to probe drivers to know their existence

29



‣ Essentials‣ Marvell SoC abstract bus (mvsoc)

‣ Generic bus for all Marvell platforms‣ Unified autoconfiguration process‣ Unified chip-detection‣ Common resource management‣ Was already in the tree

‣ Difficult to maintain for a large number of platforms

30



‣ Device drivers‣ Reused in-tree drivers for:

• Timers (with significant changes)• Serial IF (added support for ns16750)• PCIe (mvpex with adjustments)• USB (with changes)• SATA• RTC• I2C (with changes)

‣ Developed

• Network Accelerator• SPI (for Flash)

31



‣ Validation‣ Testing with ATF (Automated Testing Framework)

‣ Simple way for overall system reliability check‣ atf-run | atf-report and wait (then read log)‣ Helped in finding undefined instructions in libraries‣ 2904 passed test cases‣ 71 failed test cases (mostly known issues for that baseline)

‣ Testing suites:

‣ LMbench‣ iperf

‣ Long runs of distribution buils

32


Integration to mainline NetBSD

‣ Changes in the current development branch‣ Fixed bugs with atomic operations

‣ Unified and simplified initarm()

‣ NetBSD/arm community remarks

‣ Upstreamed features:‣ Armada XP essentials

‣ ns16750 support

‣ SPI driver

‣ Bug fixes to: spiflash.c, mvpex.c

33


Integration to mainline NetBSD

‣ Not in the mainline:‣ Networking driver

‣ I/O cache coherency support

34


Comparison with the mainline FreeBSD

● SMP support (quad core operations)● FDT support● Superpages support● Existing device drivers:

● MPIC● Timers● UART● PCIe● SATA● USB 2.0● I2C

● UP only● Hard-coded device descriptions

● Existing device drivers:● MPIC● Timers● UART● PCIe● SATA● USB 2.0● I2C● SDIO● SPI

35


Acknowledgements

‣ Mentors of the project

‣ Rafał Jaworowski (Semihalf, The FreeBSD Project)

‣ Bartłomiej Sięka (Semihalf)

‣ People involved

‣ Grzegorz Bernacki

‣ Michał Dubiel

‣ Grzegorz Jaszczyk

‣ Pablo Ribalta

‣ Marcin Wojtas

36


Acknowledgements

Special thanks to

‣ Radosław Kujawa (NetBSD project)

‣ Shadi Ammouri (Marvell)

37


Any questions?

38

NetBSD on Marvell Armada XP System on a Chip - Semihalf · NetBSD on Marvell Armada XP System on a...

Documents

Transcript of NetBSD on Marvell Armada XP System on a Chip - Semihalf · NetBSD on Marvell Armada XP System on a...