0DVWHULQJ$UGXLQR-VRQ - ArduinoJson: Efficient JSON ...Contents Contents iv 1 Introduction 1 1.1...

Mastering ArduinoJson Efficient JSON serialization for embedded C++

BENOIT BLANCHON CREATOR OF ARDUINOJSON

Contents

Contents iv

1 Introduction 11.1 About this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Introduction to JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2.1 What is JSON? . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2.2 What is serialization? . . . . . . . . . . . . . . . . . . . . . . . 41.2.3 What can you do with JSON? . . . . . . . . . . . . . . . . . . 41.2.4 History of JSON . . . . . . . . . . . . . . . . . . . . . . . . . . 51.2.5 Why is JSON so popular? . . . . . . . . . . . . . . . . . . . . . 61.2.6 The JSON syntax . . . . . . . . . . . . . . . . . . . . . . . . . 71.2.7 Binary data in JSON . . . . . . . . . . . . . . . . . . . . . . . 10

1.3 Introduction to ArduinoJson . . . . . . . . . . . . . . . . . . . . . . . 121.3.1 What ArduinoJson is . . . . . . . . . . . . . . . . . . . . . . . 121.3.2 What ArduinoJson is not . . . . . . . . . . . . . . . . . . . . . 121.3.3 What makes ArduinoJson different? . . . . . . . . . . . . . . . 131.3.4 Does size really matter? . . . . . . . . . . . . . . . . . . . . . . 151.3.5 What are the alternatives to ArduinoJson? . . . . . . . . . . . . 161.3.6 How to install ArduinoJson . . . . . . . . . . . . . . . . . . . . 171.3.7 The examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2 The missing C++ course 252.1 Why a C++ course? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262.2 Stack, heap, and globals . . . . . . . . . . . . . . . . . . . . . . . . . 28

2.2.1 Globals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282.2.2 Heap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.2.3 Stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

2.3 Pointers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332.3.1 What is a pointer? . . . . . . . . . . . . . . . . . . . . . . . . 332.3.2 Dereferencing a pointer . . . . . . . . . . . . . . . . . . . . . . 332.3.3 Pointers and arrays . . . . . . . . . . . . . . . . . . . . . . . . 34

Contents v

2.3.4 Taking the address of a variable . . . . . . . . . . . . . . . . . 352.3.5 Pointer to class and struct . . . . . . . . . . . . . . . . . . . 352.3.6 Pointer to constant . . . . . . . . . . . . . . . . . . . . . . . . 372.3.7 The null pointer . . . . . . . . . . . . . . . . . . . . . . . . . . 382.3.8 Why use pointers? . . . . . . . . . . . . . . . . . . . . . . . . . 39

2.4 Memory management . . . . . . . . . . . . . . . . . . . . . . . . . . . 402.4.1 malloc() and free() . . . . . . . . . . . . . . . . . . . . . . . . 402.4.2 new and delete . . . . . . . . . . . . . . . . . . . . . . . . . . 402.4.3 Smart pointers . . . . . . . . . . . . . . . . . . . . . . . . . . . 412.4.4 RAII . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

2.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442.5.1 What is a reference? . . . . . . . . . . . . . . . . . . . . . . . 442.5.2 Differences with pointers . . . . . . . . . . . . . . . . . . . . . 442.5.3 Reference to constant . . . . . . . . . . . . . . . . . . . . . . . 452.5.4 Rules of references . . . . . . . . . . . . . . . . . . . . . . . . 462.5.5 Common problems . . . . . . . . . . . . . . . . . . . . . . . . 462.5.6 Usage for references . . . . . . . . . . . . . . . . . . . . . . . . 47

2.6 Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482.6.1 How are the strings stored? . . . . . . . . . . . . . . . . . . . . 482.6.2 String literals in RAM . . . . . . . . . . . . . . . . . . . . . . . 482.6.3 String literals in Flash . . . . . . . . . . . . . . . . . . . . . . . 492.6.4 Pointer to the “globals” section . . . . . . . . . . . . . . . . . . 502.6.5 Mutable string in “globals” . . . . . . . . . . . . . . . . . . . . 512.6.6 A copy in the stack . . . . . . . . . . . . . . . . . . . . . . . . 522.6.7 A copy in the heap . . . . . . . . . . . . . . . . . . . . . . . . 532.6.8 A word about the String class . . . . . . . . . . . . . . . . . . 542.6.9 Pass strings to functions . . . . . . . . . . . . . . . . . . . . . 55

2.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

3 Deserialize with ArduinoJson 583.1 The example of this chapter . . . . . . . . . . . . . . . . . . . . . . . 593.2 Parse a JSON object . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

3.2.1 The JSON document . . . . . . . . . . . . . . . . . . . . . . . 603.2.2 Place the JSON document in memory . . . . . . . . . . . . . . 613.2.3 Introducing JsonBuffer . . . . . . . . . . . . . . . . . . . . . . 613.2.4 How to specify the capacity of the buffer? . . . . . . . . . . . . 623.2.5 How to determine the capacity of the buffer? . . . . . . . . . . 623.2.6 StaticJsonBuffer or DynamicJsonBuffer? . . . . . . . . . . . . . 633.2.7 Parse the object . . . . . . . . . . . . . . . . . . . . . . . . . . 643.2.8 Verify that parsing succeeds . . . . . . . . . . . . . . . . . . . 64

Contents vi

3.3 Extract values from an object . . . . . . . . . . . . . . . . . . . . . . . 663.3.1 Extract values . . . . . . . . . . . . . . . . . . . . . . . . . . . 663.3.2 Explicit casts . . . . . . . . . . . . . . . . . . . . . . . . . . . 663.3.3 Using get<T>() . . . . . . . . . . . . . . . . . . . . . . . . . . 673.3.4 When values are missing . . . . . . . . . . . . . . . . . . . . . 673.3.5 Change the default value . . . . . . . . . . . . . . . . . . . . . 68

3.4 Inspect an unknown object . . . . . . . . . . . . . . . . . . . . . . . . 703.4.1 Enumerate the keys . . . . . . . . . . . . . . . . . . . . . . . . 703.4.2 Detect the type of a value . . . . . . . . . . . . . . . . . . . . 713.4.3 Variant types and C++ types . . . . . . . . . . . . . . . . . . . 723.4.4 Test if a key exists in an object . . . . . . . . . . . . . . . . . . 73

3.5 Parse a JSON array . . . . . . . . . . . . . . . . . . . . . . . . . . . . 743.5.1 The JSON document . . . . . . . . . . . . . . . . . . . . . . . 743.5.2 Parse the array . . . . . . . . . . . . . . . . . . . . . . . . . . 753.5.3 The ArduinoJson Assistant . . . . . . . . . . . . . . . . . . . . 77

3.6 Extract values from an array . . . . . . . . . . . . . . . . . . . . . . . 783.6.1 Unrolling the array . . . . . . . . . . . . . . . . . . . . . . . . 783.6.2 Alternative syntaxes . . . . . . . . . . . . . . . . . . . . . . . . 783.6.3 When complex values are missing . . . . . . . . . . . . . . . . . 79

3.7 Inspect an unknown array . . . . . . . . . . . . . . . . . . . . . . . . . 813.7.1 Capacity of JsonBuffer for an unknown input . . . . . . . . . . 813.7.2 Number of elements in an array . . . . . . . . . . . . . . . . . 813.7.3 Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 823.7.4 Detect the type of the elements . . . . . . . . . . . . . . . . . 83

3.8 The zero-copy mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . 843.8.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 843.8.2 An example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 843.8.3 Input buffer must stay in memory . . . . . . . . . . . . . . . . 86

3.9 Parse from read-only memory . . . . . . . . . . . . . . . . . . . . . . . 873.9.1 The example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 873.9.2 Duplication is required . . . . . . . . . . . . . . . . . . . . . . 873.9.3 Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 883.9.4 Other types of read-only input . . . . . . . . . . . . . . . . . . 89

3.10 Parse from stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 913.10.1 Parse from a file . . . . . . . . . . . . . . . . . . . . . . . . . . 913.10.2 Parse from an HTTP response . . . . . . . . . . . . . . . . . . 92

4 Serialize with ArduinoJson 974.1 The example of this chapter . . . . . . . . . . . . . . . . . . . . . . . 98

Contents vii

4.2 Create an object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 994.2.1 The example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 994.2.2 Allocate the JsonBuffer . . . . . . . . . . . . . . . . . . . . . . 994.2.3 Create the object . . . . . . . . . . . . . . . . . . . . . . . . . 1004.2.4 Add the values . . . . . . . . . . . . . . . . . . . . . . . . . . . 1004.2.5 Second syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . 1014.2.6 Third syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1014.2.7 Replace values . . . . . . . . . . . . . . . . . . . . . . . . . . . 1024.2.8 Remove values . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

4.3 Create an array . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1034.3.1 The example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1034.3.2 Allocate the JsonBuffer . . . . . . . . . . . . . . . . . . . . . . 1034.3.3 Create the array . . . . . . . . . . . . . . . . . . . . . . . . . . 1034.3.4 Add values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1044.3.5 Replace values . . . . . . . . . . . . . . . . . . . . . . . . . . . 1044.3.6 Remove values . . . . . . . . . . . . . . . . . . . . . . . . . . . 1054.3.7 Add null . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1054.3.8 Add pre-formatted JSON . . . . . . . . . . . . . . . . . . . . . 106

4.4 Serialize to memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1074.4.1 Minified JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . 1074.4.2 Specify (or not) the size of the output buffer . . . . . . . . . . 1074.4.3 Prettified JSON . . . . . . . . . . . . . . . . . . . . . . . . . . 1084.4.4 Compute the length . . . . . . . . . . . . . . . . . . . . . . . . 1084.4.5 Serialize to a String . . . . . . . . . . . . . . . . . . . . . . . . 1094.4.6 Cast a JsonVariant to a String . . . . . . . . . . . . . . . . . . 110

4.5 Serialize to stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1114.5.1 What’s an output stream? . . . . . . . . . . . . . . . . . . . . 1114.5.2 Serialize to Serial . . . . . . . . . . . . . . . . . . . . . . . . . 1124.5.3 Serialize to a file . . . . . . . . . . . . . . . . . . . . . . . . . . 1124.5.4 Serialize to an HTTP request . . . . . . . . . . . . . . . . . . . 113

4.6 Duplication of strings . . . . . . . . . . . . . . . . . . . . . . . . . . . 1174.6.1 An example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1174.6.2 Copy only occurs when adding values . . . . . . . . . . . . . . 1184.6.3 Why copying Flash strings? . . . . . . . . . . . . . . . . . . . . 1184.6.4 RawJson() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

5 Inside ArduinoJson 1205.1 Why JsonBuffer? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

5.1.1 Memory representation . . . . . . . . . . . . . . . . . . . . . . 1215.1.2 Dynamic memory . . . . . . . . . . . . . . . . . . . . . . . . . 122

Contents viii

5.1.3 Memory pool . . . . . . . . . . . . . . . . . . . . . . . . . . . 1235.1.4 Strengths and weaknesses . . . . . . . . . . . . . . . . . . . . . 124

5.2 Inside StaticJsonBuffer . . . . . . . . . . . . . . . . . . . . . . . . . . 1255.2.1 Fixed capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . 1255.2.2 Compile-time determination . . . . . . . . . . . . . . . . . . . . 1255.2.3 Stack memory . . . . . . . . . . . . . . . . . . . . . . . . . . . 1265.2.4 Limitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1265.2.5 Other usages . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1275.2.6 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 1275.2.7 Step by step . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

5.3 Inside DynamicJsonBuffer . . . . . . . . . . . . . . . . . . . . . . . . . 1305.3.1 Chunks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1305.3.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1305.3.3 Step by step . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1315.3.4 Comparison with StaticJsonBuffer . . . . . . . . . . . . . . . . 1315.3.5 How to choose? . . . . . . . . . . . . . . . . . . . . . . . . . . 132

5.4 Inside JsonArray . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1335.4.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 1335.4.2 Creating a JsonArray . . . . . . . . . . . . . . . . . . . . . . . 1335.4.3 Parsing a JsonArray . . . . . . . . . . . . . . . . . . . . . . . . 1345.4.4 Invalid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1345.4.5 Copying a JsonArray . . . . . . . . . . . . . . . . . . . . . . . . 1345.4.6 JsonArray as a generic container . . . . . . . . . . . . . . . . . 1345.4.7 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

5.5 Inside JsonObject . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1375.5.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 1375.5.2 Creating a JsonObject . . . . . . . . . . . . . . . . . . . . . . . 1375.5.3 Parsing a JsonObject . . . . . . . . . . . . . . . . . . . . . . . 1385.5.4 Invalid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1385.5.5 Copying a JsonObject . . . . . . . . . . . . . . . . . . . . . . . 1385.5.6 JsonObject as a generic container . . . . . . . . . . . . . . . . . 1395.5.7 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1395.5.8 Remark on operator[] . . . . . . . . . . . . . . . . . . . . . . 141

5.6 Inside JsonVariant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1425.6.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 1425.6.2 Undefined . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1435.6.3 The unsigned long trick . . . . . . . . . . . . . . . . . . . . . . 1435.6.4 ArduinoJson’s configuration . . . . . . . . . . . . . . . . . . . . 1445.6.5 Iterating through a JsonVariant . . . . . . . . . . . . . . . . . . 1455.6.6 The or operator . . . . . . . . . . . . . . . . . . . . . . . . . . 147

Contents ix

5.6.7 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1475.7 Inside the parser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

5.7.1 Invoke the parser . . . . . . . . . . . . . . . . . . . . . . . . . 1495.7.2 Two modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1505.7.3 Nesting limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1515.7.4 Quotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1525.7.5 Escape sequences . . . . . . . . . . . . . . . . . . . . . . . . . 1535.7.6 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1545.7.7 Stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

5.8 Inside the serializer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1555.8.1 Invoke the serializer . . . . . . . . . . . . . . . . . . . . . . . . 1555.8.2 Measure the length . . . . . . . . . . . . . . . . . . . . . . . . 1565.8.3 Escape sequences . . . . . . . . . . . . . . . . . . . . . . . . . 1565.8.4 Float to string . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

5.9 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1585.9.1 The ArduinoJson namespace . . . . . . . . . . . . . . . . . . . 1585.9.2 JsonBuffer::clear() . . . . . . . . . . . . . . . . . . . . . . . . 1585.9.3 Code coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . 1595.9.4 Fuzzing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1595.9.5 Portability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1605.9.6 Online compiler . . . . . . . . . . . . . . . . . . . . . . . . . . 1615.9.7 License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

6 Troubleshooting 1636.1 Program crashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164

6.1.1 Undefined Behaviors . . . . . . . . . . . . . . . . . . . . . . . . 1646.1.2 A bug in ArduinoJson? . . . . . . . . . . . . . . . . . . . . . . 1646.1.3 Null string . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1656.1.4 Use after free . . . . . . . . . . . . . . . . . . . . . . . . . . . 1656.1.5 Return of stack variable address . . . . . . . . . . . . . . . . . 1676.1.6 Buffer overflow . . . . . . . . . . . . . . . . . . . . . . . . . . 1696.1.7 Stack overflow . . . . . . . . . . . . . . . . . . . . . . . . . . . 1706.1.8 How to detect these bugs? . . . . . . . . . . . . . . . . . . . . 171

6.2 Deserialization issues . . . . . . . . . . . . . . . . . . . . . . . . . . . 1736.2.1 A lack of information . . . . . . . . . . . . . . . . . . . . . . . 1736.2.2 Is input valid? . . . . . . . . . . . . . . . . . . . . . . . . . . . 1746.2.3 Is the JsonBuffer big enough? . . . . . . . . . . . . . . . . . . 1746.2.4 Is there enough RAM? . . . . . . . . . . . . . . . . . . . . . . 1756.2.5 How deep is the document? . . . . . . . . . . . . . . . . . . . . 1766.2.6 The first deserialization works? . . . . . . . . . . . . . . . . . . 177

Contents x

6.3 Serialization issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1786.3.1 The JSON document is incomplete . . . . . . . . . . . . . . . . 1786.3.2 The JSON document contains garbage . . . . . . . . . . . . . . 1786.3.3 Too much duplication . . . . . . . . . . . . . . . . . . . . . . . 1806.3.4 The first serialization succeeds? . . . . . . . . . . . . . . . . . . 181

6.4 Understand compiler errors . . . . . . . . . . . . . . . . . . . . . . . . 1826.4.1 Long compiler errors . . . . . . . . . . . . . . . . . . . . . . . 1826.4.2 How GCC presents errors . . . . . . . . . . . . . . . . . . . . . 1836.4.3 The first error in our example . . . . . . . . . . . . . . . . . . . 1856.4.4 The second error in our example . . . . . . . . . . . . . . . . . 186

6.5 Common error messages . . . . . . . . . . . . . . . . . . . . . . . . . . 1896.5.1 Ambiguous overload for operator= . . . . . . . . . . . . . . . . 1896.5.2 Conversion from const char* to char* . . . . . . . . . . . . . . 1896.5.3 Conversion from const char* to int . . . . . . . . . . . . . . . 1906.5.4 equals is not a member of StringTraits<const int&> . . . . . . 1916.5.5 Undefined reference to __cxa_guard_acquire and

__cxa_guard_release . . . . . . . . . . . . . . . . . . . . . . . . 1926.6 Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

6.6.1 The problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1946.6.2 Print decorator . . . . . . . . . . . . . . . . . . . . . . . . . . 1946.6.3 Stream decorator . . . . . . . . . . . . . . . . . . . . . . . . . . 196

6.7 Ask for help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

7 Case Studies 2007.1 Configuration in SPIFFS . . . . . . . . . . . . . . . . . . . . . . . . . 201

7.1.1 Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2017.1.2 The JSON document . . . . . . . . . . . . . . . . . . . . . . . 2017.1.3 The configuration class . . . . . . . . . . . . . . . . . . . . . . 2027.1.4 load() and save() members . . . . . . . . . . . . . . . . . . . . 2037.1.5 Save an ApConfig into a JsonObject . . . . . . . . . . . . . . . 2047.1.6 Load an ApConfig from a JsonObject . . . . . . . . . . . . . . . 2047.1.7 Safely copy strings from JsonObject . . . . . . . . . . . . . . . 2047.1.8 Save a Config to a JsonObject . . . . . . . . . . . . . . . . . . 2057.1.9 Load a Config from a JsonObject . . . . . . . . . . . . . . . . . 2067.1.10 Save configuration to a file . . . . . . . . . . . . . . . . . . . . 2077.1.11 Read configuration from a file . . . . . . . . . . . . . . . . . . 2087.1.12 Choosing the JsonBuffer . . . . . . . . . . . . . . . . . . . . . 2097.1.13 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

7.2 OpenWeatherMap on mkr1000 . . . . . . . . . . . . . . . . . . . . . . 2117.2.1 Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211

Contents xi

7.2.2 OpenWeatherMap’s API . . . . . . . . . . . . . . . . . . . . . 2117.2.3 The JSON response . . . . . . . . . . . . . . . . . . . . . . . . 2127.2.4 Reducing memory usage . . . . . . . . . . . . . . . . . . . . . 2137.2.5 Jumping in the stream . . . . . . . . . . . . . . . . . . . . . . 2147.2.6 The code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2157.2.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216

7.3 Weather Underground on ESP8266 . . . . . . . . . . . . . . . . . . . . 2177.3.1 Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2177.3.2 Weather Underground’s API . . . . . . . . . . . . . . . . . . . 2177.3.3 HTTP client . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2187.3.4 The JSON response . . . . . . . . . . . . . . . . . . . . . . . . 2187.3.5 Reducing memory usage . . . . . . . . . . . . . . . . . . . . . 2207.3.6 Jumping in the stream . . . . . . . . . . . . . . . . . . . . . . 2217.3.7 The code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2227.3.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

7.4 JSON-RPC with Kodi . . . . . . . . . . . . . . . . . . . . . . . . . . . 2247.4.1 Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2247.4.2 JSON-RPC Request . . . . . . . . . . . . . . . . . . . . . . . . 2247.4.3 JSON-RPC Response . . . . . . . . . . . . . . . . . . . . . . . 2257.4.4 A JSON-RPC framework . . . . . . . . . . . . . . . . . . . . . 2267.4.5 JsonRpcRequest . . . . . . . . . . . . . . . . . . . . . . . . . . 2267.4.6 JsonRpcResponse . . . . . . . . . . . . . . . . . . . . . . . . . . 2287.4.7 JsonRpcClient . . . . . . . . . . . . . . . . . . . . . . . . . . . 2287.4.8 Send a notification to Kodi . . . . . . . . . . . . . . . . . . . . 2307.4.9 Get properties from Kodi . . . . . . . . . . . . . . . . . . . . . 2327.4.10 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233

7.5 Recursive analyzer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2357.5.1 Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2357.5.2 Read from the serial port . . . . . . . . . . . . . . . . . . . . . 2357.5.3 Test the type of a JsonVariant . . . . . . . . . . . . . . . . . . 2367.5.4 Print values . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2387.5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240

8 Conclusion 241

Index 242

Chapter 4Serialize with ArduinoJson

Any fool can write code that a computer can understand. Good programmerswrite code that humans can understand.– Martin Fowler, Refactoring: Improving the Design of Existing Code

Chapter 4 Serialize with ArduinoJson 98

4.1 The example of this chapter

Reading a JSON document is only half of the story; we’llnow see how to write a JSON document with ArduinoJ-son.

The previous chapter revolved around weather forecast forthe city of New York. We’ll use a very different examplefor this chapter: pushing data to Adafruit IO.

Adafruit IO is a cloud storage for IoT data. They have afree plan with the following restrictions:

• 30 data points per minutes

• 30 days of data storage

• 10 feeds

If you need more, it’s just $10 a month. The service is veryeasy to use. All you need is an Adafruit account (yes, youcan use the account from the Adafruit shop).

As we did in the previous chapter, we’ll start with a simple JSON document and addcomplexity step by step.

https://io.adafruit.com


4.2 Create an object

4.2.1 The example

Here is the JSON object we want to create:

{

"value": 42,

"lat": 48.748010,

"lon": 2.293491

}

It’s a flat object, meaning it has no nested object or array, and it contains the followingpiece of information:

1. value is an integer we want to save in Adafruit IO.

2. lat is the latitude coordinate where the value was measured.

3. lon is the longitude coordinate where the value was measured.

Adafruit IO supports other optional members (like the elevation coordinate and the timeof measurement), but the three members above are sufficient for our example.

4.2.2 Allocate the JsonBuffer

As for the deserialization, we start by creating a JsonBuffer to hold the in-memoryrepresentation of the object. The previous chapter introduces the JsonBuffer; please goback and read ”Introducing JsonBuffer” if needed, as we won’t repeat here.

We need to compute the capacity of the JsonBuffer. As the JSON document is simple,we can do the computation manually. We just have one object with no nested values, sothe capacity is JSON_OBJECT_SIZE(3). Remember that you can use ArduinoJson Assistantwhen the JSON document is more elaborate.

It’s a reasonably small JsonBuffer, which fits in the stack on any microcontroller, so wecan use a StaticJsonBuffer.

Here is the code:


const int capacity = JSON_OBJECT_SIZE(3);

StaticJsonBuffer<capacity> jb;

4.2.3 Create the object

It’s not possible to directly instantiate a JsonObject, we’ll see why in the next chapter.Instead, we need to ask the JsonBuffer to create one for us:

// Create a JsonObject

JsonObject& obj = jb.createObject();

This statement creates an empty object; its memory usage is currentlyJSON_OBJECT_SIZE(0)

As the object is located inside the JsonBuffer, we receive a reference to it. We saw thesame thing with parseObject() in the previous chapter.

The factory design patternJsonObject cannot be constructed directly, but JsonBuffer provides a methodto create an object.It is an implementation of the “factory” design pattern: JsonBuffer is afactory of JsonObject.

4.2.4 Add the values

Adding values to a JsonObject is very similar to reading them.

There are several syntaxes, but the simplest is to use the subscript operator ([]):

obj["value"] = 42;

obj["lat"] = 48.748010;

obj["lon"] = 2.293491;

https://en.wikipedia.org/wiki/Abstract_factory_pattern


The object’s memory usage is now JSON_OBJECT_SIZE(3), meaning that the JsonBuffer

is full. When the JsonBuffer is full, you cannot add more values to the object, so don’tforget to increase the capacity if you need.

4.2.5 Second syntax

With the syntax presented above, it’s not possible to tell if the insertion succeeded orfailed. Let’s see another syntax:

obj.set("value", 42);

obj.set("lat", 48.748010);

obj.set("lon", 2.293491);

The executable generated by the compiler is the same as with the previous syntax,except that you can check if the insertion succeeded. You can check the result ofJsonObject::set(), which will be true for success or false for failure. Again, insertionfails if the JsonBuffer is full.

Personally, I never check if insertion succeeds in my programs. The reason is simple:the JSON document is roughly the same for each iteration; if it works once, it alwaysworks. There is no reason to bloat the code for a situation that cannot happen.

4.2.6 Third syntax

The syntax we just saw used JsonObject::set(). We can combine both syntax by usingJsonVariant::set():

obj["value"].set(42);

obj["lat"].set(48.748010);

obj["lon"].set(2.293491);

Again, the compiled executable is the same, and you can check the return value too.


4.2.7 Replace values

It’s possible to replace a value in the object, for example:

obj["value"] = 42;

obj["value"] = 43;

It doesn’t require a new allocation in the JsonBuffer, so if the first insertion succeeds,the second will succeed too.

4.2.8 Remove values

It’s possible to erase values from an object by calling JsonObject::remove(key). How-ever, for reasons that will become clear in the next chapter, this function doesn’t releasethe memory in the JsonBuffer.

The remove() function is a frequent cause of bugs because it creates a memory leak inthe program. Indeed, if you add and remove values in a loop, the JsonBuffer grows,but memory is never released.

Code smellIn practice, this problem only happens in programs that use ArduinoJsonto store the state of the application, which is not what ArduinoJson is for.Trying to optimize this use-case would inevitably impact the size and speedof ArduinoJson. Be careful not to fall into this common anti-pattern andmake sure you read the case studies to see how ArduinoJson should be used.


4.3 Create an array

4.3.1 The example

Our next step will be to construct an array containing two objects:

[

{

"key": "a1",

"value": 12

},

{

"key": "a2",

"value": 34

}

]

4.3.2 Allocate the JsonBuffer

As usual, we start by computing the capacity of the JsonBuffer:

• There is one array with two elements: JSON_ARRAY_SIZE(2)

• There are two objects with two pairs: 2*JSON_OBJECT_SIZE(2)

Here is the code:

const int capacity = JSON_ARRAY_SIZE(2) + 2*JSON_OBJECT_SIZE(2);


4.3.3 Create the array

We create arrays the same way we create objects, by using the JsonBuffer as a factory:

JsonArray& arr = jb.createArray();


4.3.4 Add values

To add the nested objects to the array, we could create the two objects and add themto the array like that:

JsonObject& obj1 = jb.createObject();

obj1["key"] = "a1";

obj1["value"] = analogRead(A1);

JsonObject& obj2 = jb.createObject();

obj2["key"] = "a2";


arr.add(obj1);

arr.add(obj2);

JsonArray::add() adds a new value at the end of the array. In this case, we added twoJsonObjects, but you can use any value that a JsonVariant supports: int, float…

We just saw one way to create an array of object, but we can do better. Instead, wecan call JsonArray::createNestedObject(), which creates the nested object and adds itto the end of the array. You can see on the snippet below that it leads to a simplercode:

JsonObject& obj1 = arr.createNestedObject();

obj1["key"] = "a1";


JsonObject& obj2 = arr.createNestedObject();

obj2["key"] = "a2";


Thanks to this technique, the program is shorter, faster, and more readable.

4.3.5 Replace values

As for objects, it’s possible to replace values in arrays using either JsonArray::operator[]or JsonArray::set():


arr[0] = 666;

arr[1] = 667;

Replacing the value doesn’t require a new allocation in the JsonBuffer. However, ifthere was memory hold by the previous value, for example, a JsonObject, this memoryis not released. Doing so would require counting references to the nested JsonObject,which ArduinoJson does not.

4.3.6 Remove values

As for objects, you can delete a slot of the array, by using JsonArray::remove():

arr.remove(0);

As described in the previous section, remove() doesn’t release the memory from theJsonBuffer. You should never call this function in a loop.

4.3.7 Add null

To conclude this section, let’s see how we can insert special values in the JSON docu-ment.

The first special value is null, which is a legal token in a JSON document. In Arduino-Json, it is a string whose address is zero:

// adds "null"

arr.add("null");

// add null

arr.add((char*)0);

The program above produces the following JSON document:

[

"null",

null


]

4.3.8 Add pre-formatted JSON

The other special value is a JSON element that is already formatted and that Arduino-Json should not treat as a string.

You can do that by wrapping the string with a call to RawJson():

// adds "[1,2]"

arr.add("[1,2]");

// adds [1,2]

arr.add(RawJson("[1,2]"));

The program above produces the following JSON document:

[

"[1,2]",

[

1,

2

]

]


4.4 Serialize to memory

We saw how to construct an array, and it’s time to serialize it into a JSON document.There are several ways to do that. We’ll start with a JSON document in memory.

We could use a String but, as you may now start to see, I don’t like using dynamicmemory allocation. Instead, we’d use a good old char[]:

// Declare a buffer to hold the result

char output[128];

4.4.1 Minified JSON

Suppose we’re still using our previous example for JsonArray, if we want to produce aJSON document out of it, we just need to call JsonArray::printTo():

// Produce a minified JSON document

arr.printTo(output);

Now the string output contains:

[{"key":"a1","value":12},{"key":"a2","value":34}]

As you see, there are neither space nor line breaks; it’s a “minified” JSON document.

4.4.2 Specify (or not) the size of the output buffer

If you’re a C programmer, you may have been surprised that I didn’t provide the size ofthe buffer to printTo(). Indeed, there is an overload of printTo() that takes a char*

and a size:

arr.printTo(output, sizeof(output));


But that’s not the overload we called in the previous snippet. Instead, we called atemplate method that infers the size of the buffer from its type (in our case char[128]).

Of course, this shorter syntax only works because output is an array. If it were a char*,we would have had to specify the size.

4.4.3 Prettified JSON

The minified version is what you use to store or transmit a JSON document becausethe size is optimal. However, it’s not very easy to read. Humans prefer “prettified”JSON documents with spaces and line breaks.

To produce a prettified document, you just need to use prettyPrintTo() instead ofprintTo():

// Produce a prettified JSON document

arr.prettyPrintTo(output);

Here is the output:

[

{

"key": "a1",

"value": 12

},

{

"key": "a2",

"value": 34

}

]

Of course, you need to make sure that the output buffer is big enough; otherwise theJSON document will be incomplete.

4.4.4 Compute the length

ArduinoJson allows computing the length of the JSON document before producing it.This information is useful for:


1. allocating an output buffer,

2. reserving the size on disk, or

3. setting the Content-Length header.

There are two methods, depending on the type of document you want to produce:

// Compute the length of the minified JSON document

int len1 = arr.measureLength();

// Compute the length of the prettified JSON document

int len2 = arr.measurePrettyLength();

In both cases, the return value doesn’t count the null-terminator.

By the way, printTo() and prettyPrintTo() return the number of bytes written. Theirreturn values are the same as measureLength() and measurePrettyLength(), except if theoutput buffer was too small.

Avoid prettified documentsThe sizes in the example above are 73 and 110. In this case, the prettifiedversion is only 50% bigger because the document is simple. But, in mostcase, the ratio is largely above 100%. Remember, we’re in an embeddedenvironment: every byte counts and so does every CPU cycle. Always prefera minified version.

4.4.5 Serialize to a String

The functions printTo() and prettyPrintTo() have overloads taking a String:

String output = "JSON = ";

arr.printTo(output);

The behavior is slightly different as the JSON document is appended to end the String.The snippet above sets the content of the output variable to:


JSON = [{"key":"a1","value":12},{"key":"a2","value":34}]

4.4.6 Cast a JsonVariant to a String

You should remember from the chapter on deserialization that we must cast JsonVariantto the type we want to read.

It is also possible to cast a JsonVariant to a String. If the JsonVariant contains astring, the return value is a copy of the string. However, if the JsonVariant containssomething else, the return value is a serialization of the value.

We could rewrite the previous example like this:

// Wrap the JsonArray in a JsonVariant

JsonVariant v = arr;

// Cast the JsonVariant to a string

String output = "toto" + v.as<String>();

Unfortunately, this trick is only available on JsonVariant; you cannot do the same withJsonArray and JsonObject (unless you put them in a JsonVariant). Furthermore, thistechnique only produces a minified document.


4.5 Serialize to stream

4.5.1 What’s an output stream?

In the previous section, we saw how to serialize an array or an object. For now, everyJSON document we produced remained in memory, but that’s usually not what wewant.

In many situations, it’s possible to send the JSON document directly to its destination(whether it’s a file, a serial port, or a network connection) without any copy in RAM.

We saw in the previous chapter what an input stream is, and that Arduino representsthis concept with the class Stream.

Similar to input streams, we also have “output streams,” which are sinks of bytes. Wecan write to an output stream, but we cannot read. In the Arduino land, an outputstream is materialized by the class Print.

Here are examples of classes derived from Print:

Library Class Well known instancesCore HardwareSerial Serial, Serial1…ESP8266 FS File

Ethernet EthernetClient

Ethernet EthernetUDP

GSM GSMClient

LiquidCrystal LiquidCrystal

SD File

SoftwareSerial SoftwareSerial

Wifi WifiClient

Wire TwoWire Wire

std::ostream

In the C++ Standard Library, an output stream is represented by the classstd::ostream. ArduinoJson supports both Print and std::ostream.


4.5.2 Serialize to Serial

The most famous implementation of Print is HardwareSerial, which is the class ofSerial. To serialize a JsonArray or a JsonObject to the serial port of your Arduino, justpass Serial to printTo():

// Print a minified version to the serial port

arr.printTo(Serial);

// Same with a prettified version

arr.prettyPrintTo(Serial);

You can see the result in the Arduino Serial Monitor, which is very handy for debug-ging.

There are also other serial port implementations that you can use this way, for exampleSoftwareSerial and TwoWire.

4.5.3 Serialize to a file

Similarly, we can use a File instance as the target of printTo() and prettyPrintTo().Here is an example with the SD library:


// Open file for writing

File file = SD.open("adafruit.txt", FILE_WRITE);

// Write a prettified JSON document to the file

arr.prettyPrintTo(file);

You can find the complete source code for this example in the folder WriteSdCard of thezip file.

You can apply the same technique to write a file on an ESP8266, as we’ll see in thecase studies.

4.5.4 Serialize to an HTTP request

We’re now reaching our goal of sending our measurements to Adafruit IO.

To do that, we need to send the following JSON document:

{

"location": {

"lat": 48.748010,

"lon": 2.293491

},

"feeds": [

{

"key": "a1",

"value": 42

},

{

"key": "a2",

"value": 43

}

]

}

This document contains the values to add to our two feeds a1 and a2.

We will send this document in the body of the following HTTP request:


POST /api/v2/bblanchon/groups/arduinojson/data HTTP/1.1

Host: io.adafruit.com

Connection: close

Content-Length: 103

Content-Type: application/json

X-AIO-Key: baf4f21a32f6438eb82f83c3eed3f3b3

{"location":{"lat":48.748010,"lon":2.293491},"feeds":[{"key":"a1","value"...

Here is the program:

// Allocate JsonBuffer

const int capacity = JSON_ARRAY_SIZE(2) + 4 * JSON_OBJECT_SIZE(2);


// Create JsonObject

JsonObject &root = jb.createObject();

// Add location

JsonObject &location = root.createNestedObject("location");

location["lat"] = 48.748010;

location["lon"] = 2.293491;

// Add feeds array

JsonArray &feeds = root.createNestedArray("feeds");

JsonObject &feed1 = feeds.createNestedObject();

feed1["key"] = "a1";

feed1["value"] = analogRead(A1);

JsonObject &feed2 = feeds.createNestedObject();

feed2["key"] = "a2";

feed2["value"] = analogRead(A2);

// Connect to the HTTP server

EthernetClient client;

client.setTimeout(10000);

client.connect("io.adafruit.com", 80);

// Send "POST /api/v2/bblanchon/groups/arduinojson/data HTTP/1.1"


client.print("POST /api/v2/");

client.print(username);

client.print("/groups/");

client.print(groupname);

client.println("/data HTTP/1.1");

// Send the HTTP headers

client.println("Host: io.adafruit.com");

client.println("Connection: close");

client.print("Content-Length: ");

client.println(root.measureLength());

client.println("Content-Type: application/json");

client.print("X-AIO-Key: ");

client.println(apiKey);

// Terminate headers with a blank line

client.println();

// Send JSON document in body

root.printTo(client);

You can find the complete source code of this example in the folder AdafruitIo in thezip file.

If you want to reproduce this example, you need to follow these steps:

1. Create an account on Adafruit IO (a free tier is sufficient).

2. Create a feed group named arduinojson.

3. In that group, create two feeds a1 and a2.

4. Set the constant username to your user name.

5. Set the constant apiKey to your AIO key.


Below is a picture of a dashboard showing the data from this example.


4.6 Duplication of strings

When you add a string value to a JsonArray or a JsonObject, ArduinoJson either storesa pointer or a copy of the string depending on its type. If the string is a const char*,it stores a pointer; otherwise, it makes a copy.

String type Storageconst char* pointerchar* copyString copyconst __FlashStringHelper* copy

As usual, the copy lives in the JsonBuffer, so you may need to increase its capacitydepending on the type of string you use.

The table above reflects the new rules that are in place since ArduinoJson 5.13; on olderversions char* were stored with a pointer, which caused surprising effects.

4.6.1 An example

Compare this program:

// Create the array ["value1","value2"]


arr.add("value1");

arr.add("value2");

// Print the memory usage

Serial.println(jb.size());

with the following:

// Create the array ["value1","value2"]


arr.add(String("value1"));

arr.add(String("value2"));


// Print the memory usage

Serial.println(jb.size());

They both produce the same JSON document, but the second one require much morememory because ArduinoJson has to make copies. If you run these programs on anATmega328, you’ll see 20 for the first one and 32 for the second.

4.6.2 Copy only occurs when adding values

In the example above, ArduinoJson copied the Strings because it needed to add themto the JsonArray. On the other hand, if you use a String to extract a value from aJsonObject, it doesn’t make a copy.

Here is an example:

JsonObject& obj = jb.createObject();

// The following line produces a copy of "key"

obj[String("key")] = "value";

// The following line produces no copy

const char* value = obj[String("key")];

4.6.3 Why copying Flash strings?

I understand that it is disappointing that ArduinoJson copies Flash strings into theJsonBuffer. Unfortunately, there are several situations where it needs to have thestrings in RAM.

For example, if the user calls JsonVariant::as<char*>(), a pointer to the copy is re-turned:

// The value is originally in Flash memory

obj["hello"] = F("world");

// But the returned value is in RAM (in the JsonBuffer)


const char* world = obj["hello"];

It is required for JsonPair too. If the string is a key in an object and the user iteratesthrough the object, the JsonPair contains a pointer to the copy:

// The key is originally in Flash memory

obj[F("hello")] = "world";

for(JsonPair& kvp : obj) {

// But the key is actually stored in RAM (in the JsonBuffer)

const char* key = kvp.key;

}

However, retrieving a value using a Flash string as a key doesn’t cause a copy:

// The Flash string is not copied in this case

const char* world = obj[F("hello")];

Avoid Flash string with ArduinoJsonStoring strings in Flash is a great way to reduce RAM usage, but rememberthat they are copied into the JsonBuffer. If you wrap all your strings withF(), you’ll need a much bigger JsonBuffer. Moreover, the program will wastea lot of time copying the string; it will be much slower than with conventionalstrings.

4.6.4 RawJson()

We saw earlier in this chapter that the RawJson() function marks strings as JSONsegments that should not be treated as string values.

Since ArduinoJson 5.13, RawJson() supports all the string types (char*, const char*,String and const __FlashStringHelper*) and obeys to the rules stated in the tableabove.

Continue reading...

That was a free chapter from “Mastering ArduinoJson”; the book contains seven chap-ters like this one. Here is what readers say:

This book is 100% worth it. Between solving my immediate problem inminutes, Chapter 2, and the various other issues this book made solvingeasy, it is totally worth it. I build software but I work in managed languagesand for someone just getting started in C++and embedded programming thisbook has been indispensable. — Nathan Burnett

I think the missing C++course and the troubleshooting chapter are worththe money by itself. Very useful for C programming dinosaurs like myself.

— Doug Petican

The short C++section was a great refresher. The practical use of Arduino-Json in small embedded processors was just what I needed for my homeautomation work. Certainly worth having! Thank you for both the bookand the library. — Douglas S. Basberg

For a really reasonable price, not only you’ll learn new skills, but you’ll also be one ofthe few people that contribute to sustainable open-source software. Yes, givingmoney for free software is a political act!

The e-book comes in three formats: PDF, epub and mobi. If you purchase the e-book,you get access to newer versions for free. A carefully edited paperback edition isalso available.

Ready to jump in?Go to arduinojson.org/book and use the coupon code THIRTY to get a 30% discount.

https://arduinojson.org/book/

0DVWHULQJ$UGXLQR-VRQ - ArduinoJson: Efficient JSON ...Contents Contents iv 1 Introduction 1 1.1...

Documents

Transcript of 0DVWHULQJ$UGXLQR-VRQ - ArduinoJson: Efficient JSON ...Contents Contents iv 1 Introduction 1 1.1...