bump moc

[chenyan-dfinity]

• moc 0.13 timeout
• dfx 22, moc 0.12.1 works

[github-actions]

Note
Diffing the performance result against the published result from main branch.
Unchanged benchmarks are omitted.

SHA-2

	binary_size	SHA-256	SHA-512	account_id	neuron_id
Motoko	199_379 ($\textcolor{red}{2.94\%}$)	282_867_522 ($\textcolor{red}{5.65\%}$)	262_958_038 ($\textcolor{red}{6.10\%}$)	34_374 ($\textcolor{red}{2.19\%}$)	25_343 ($\textcolor{red}{3.31\%}$)
Rust	541_114 ($\textcolor{green}{-0.14\%}$)	82_781_801	56_787_280	41_044	39_949

Certified map

	binary_size	generate 10k	max mem	inc	witness	upgrade
Motoko	248_133 ($\textcolor{red}{1.84\%}$)	3_778_873_966 ($\textcolor{green}{-19.01\%}$)	2_741_676 ($\textcolor{green}{-20.07\%}$)	539_089 ($\textcolor{green}{-2.63\%}$)	392_680 ($\textcolor{green}{-3.74\%}$)	225_075_238 ($\textcolor{green}{-17.99\%}$)
Rust	581_838 ($\textcolor{red}{0.24\%}$)	5_014_324_957 ($\textcolor{green}{-21.77\%}$)	2_031_616 ($\textcolor{green}{-8.82\%}$)	603_110 ($\textcolor{green}{-40.89\%}$)	362_545 ($\textcolor{red}{21.58\%}$)	4_681_434_598 ($\textcolor{green}{-22.31\%}$)

Statistics

• binary_size: 1.22% [-0.47%, 2.91%]
• max_mem: -14.45% [-49.95%, 21.05%]
• cycles: -7.46% [-16.35%, 1.43%]

Basic DAO

	binary_size	init	transfer_token	submit_proposal	vote_proposal	upgrade
Motoko	278_818 ($\textcolor{red}{1.78\%}$)	513_205 ($\textcolor{red}{0.47\%}$)	23_353 ($\textcolor{red}{4.65\%}$)	19_189 ($\textcolor{red}{3.18\%}$)	20_430 ($\textcolor{red}{3.90\%}$)	161_803 ($\textcolor{red}{2.43\%}$)
Rust	848_284 ($\textcolor{red}{0.09\%}$)	507_124 ($\textcolor{red}{0.13\%}$)	88_983 ($\textcolor{red}{0.05\%}$)	117_257 ($\textcolor{green}{-0.09\%}$)	112_345 ($\textcolor{green}{-0.00\%}$)	1_483_458 ($\textcolor{red}{0.01\%}$)

DIP721 NFT

	binary_size	init	mint_token	transfer_token	upgrade
Motoko	224_993 ($\textcolor{red}{2.08\%}$)	482_026 ($\textcolor{red}{0.18\%}$)	31_119 ($\textcolor{red}{3.03\%}$)	8_891 ($\textcolor{red}{1.45\%}$)	91_908 ($\textcolor{red}{2.31\%}$)
Rust	879_204 ($\textcolor{green}{-0.08\%}$)	203_040 ($\textcolor{green}{-0.23\%}$)	306_272 ($\textcolor{red}{1.06\%}$)	71_984 ($\textcolor{red}{1.05\%}$)	1_625_825 ($\textcolor{green}{-0.25\%}$)

Statistics

• binary_size: 0.97% [-0.35%, 2.29%]
• max_mem: no change
• cycles: 1.30% [0.65%, 1.94%]

Heartbeat

	binary_size	heartbeat
Motoko	142_342 ($\textcolor{red}{3.76\%}$)	27_506 ($\textcolor{red}{41.01\%}$)
Rust	23_850 ($\textcolor{red}{0.82\%}$)	1_112 ($\textcolor{red}{131.67\%}$)

Timer

	binary_size	setTimer	cancelTimer
Motoko	150_155 ($\textcolor{red}{3.11\%}$)	56_189 ($\textcolor{red}{8.52\%}$)	4_702 ($\textcolor{red}{1.64\%}$)
Rust	504_099 ($\textcolor{red}{0.27\%}$)	63_379	11_676

Statistics

• binary_size: 1.69% [-7.30%, 10.68%]
• max_mem: no change
• cycles: 5.08% [-16.63%, 26.79%]

Garbage Collection

	generate 700k	max mem	batch_get 50	batch_put 50	batch_remove 50
default	1_074_136_336 ($\textcolor{red}{0.56\%}$)	47_793_792	119	119	119
copying	1_074_136_218 ($\textcolor{red}{0.56\%}$)	47_793_792	1_073_873_789 ($\textcolor{red}{0.56\%}$)	1_073_954_095 ($\textcolor{red}{0.56\%}$)	1_073_875_311 ($\textcolor{red}{0.56\%}$)
compacting	1_554_238_605 ($\textcolor{red}{0.56\%}$)	47_793_792	1_200_791_965 ($\textcolor{red}{0.73\%}$)	1_424_078_246 ($\textcolor{red}{0.61\%}$)	1_447_969_756 ($\textcolor{red}{0.60\%}$)
generational	2_326_734_591 ($\textcolor{red}{0.98\%}$)	47_802_256	899_105_682 ($\textcolor{red}{1.92\%}$)	1_214_812 ($\textcolor{red}{0.30\%}$)	1_107_099 ($\textcolor{red}{0.32\%}$)
incremental	29_505_471 ($\textcolor{red}{0.01\%}$)	976_097_724 ($\textcolor{red}{0.00\%}$)	469_026_873 ($\textcolor{green}{-0.61\%}$)	496_491_319 ($\textcolor{green}{-0.20\%}$)	1_282_778_770 ($\textcolor{red}{5.03\%}$)

Actor class

	binary size	put new bucket	put existing bucket	get
Map	421_218 ($\textcolor{red}{40.77\%}$)	757_731 ($\textcolor{green}{-6.86\%}$)	16_360 ($\textcolor{red}{1.52\%}$)	16_930 ($\textcolor{red}{1.62\%}$)

Statistics

• binary_size: no change
• max_mem: 0.00%
• cycles: 2.39% [-1.01%, 5.78%]

Publisher & Subscriber

	pub_binary_size	sub_binary_size	subscribe_caller	subscribe_callee	publish_caller	publish_callee
Motoko	165_900 ($\textcolor{red}{2.86\%}$)	150_198 ($\textcolor{red}{3.06\%}$)	32_864 ($\textcolor{red}{14.94\%}$)	12_214 ($\textcolor{red}{2.10\%}$)	27_075 ($\textcolor{red}{18.47\%}$)	6_629 ($\textcolor{red}{2.84\%}$)
Rust	538_668 ($\textcolor{red}{0.51\%}$)	577_244 ($\textcolor{red}{0.60\%}$)	57_488	37_836 ($\textcolor{red}{0.10\%}$)	71_326 ($\textcolor{green}{-0.03\%}$)	42_449

Statistics

• binary_size: 1.76% [0.12%, 3.39%]
• max_mem: no change
• cycles: 6.40% [-0.29%, 13.09%]

Overall Statistics

• binary_size: 1.37% [0.76%, 1.97%]
• max_mem: -9.63% [-26.59%, 7.33%]
• cycles: 0.55% [-1.73%, 2.83%]

[github-actions]

Note
The flamegraph link only works after you merge.
Unchanged benchmarks are omitted.

Cryptographic libraries

Measure different cryptographic libraries written in both Motoko and Rust.

• SHA-2 benchmarks
  • SHA-256/SHA-512. Compute the hash of a 1M Wasm binary.
  • account_id. Compute the ledger account id from principal, based on SHA-224.
  • neuron_id. Compute the NNS neuron id from principal, based on SHA-256.
• Certified map. Merkle Tree for storing key-value pairs and generate witness according to the IC Interface Specification.
  • generate 10k. Insert 10k 7-character word as both key and value into the certified map.
  • max mem. For Motoko, it reports rts_max_heap_size after generate call; For Rust, it reports the Wasm's memory page * 32Kb.
  • inc. Increment a counter and insert the counter value into the map.
  • witness. Generate the root hash and a witness for the counter.
  • upgrade. Upgrade the canister with the same Wasm. In Motoko, we use stable variable. In Rust, we convert the tree to a vector before serialization.

SHA-2

	binary_size	SHA-256	SHA-512	account_id	neuron_id
Motoko	199_379	282_867_522	262_958_038	34_374	25_343
Rust	541_114	82_781_801	56_787_280	41_044	39_949

Certified map

	binary_size	generate 10k	max mem	inc	witness	upgrade
Motoko	248_133	3_778_873_966	2_741_676	539_089	392_680	225_075_238
Rust	581_838	5_014_324_957	2_031_616	603_110	362_545	4_681_434_598

Environment

• dfx 0.23.0
• Motoko compiler 0.13.0 (source 30y2saap-vdivk61d-vl913kh2-srcdk17g)
• rustc 1.76.0 (07dca489a 2024-02-04)
• ic-repl 0.7.5
• ic-wasm 0.7.0

Sample Dapps

Measure the performance of some typical dapps:

• Basic DAO,
  with heartbeat disabled to make profiling easier. We have a separate benchmark to measure heartbeat performance.
• DIP721 NFT

Note

• The cost difference is mainly due to the Candid serialization cost.
• Motoko statically compiles/specializes the serialization code for each method, whereas in Rust, we use serde to dynamically deserialize data based on data on the wire.
• We could improve the performance on the Rust side by using parser combinators. But it is a challenge to maintain the ergonomics provided by serde.
• For real-world applications, we tend to send small data for each endpoint, which makes the Candid overhead in Rust tolerable.

Basic DAO

	binary_size	init	transfer_token	submit_proposal	vote_proposal	upgrade
Motoko	278_818	513_205	23_353	19_189	20_430	161_803
Rust	848_284	507_124	88_983	117_257	112_345	1_483_458

DIP721 NFT

	binary_size	init	mint_token	transfer_token	upgrade
Motoko	224_993	482_026	31_119	8_891	91_908
Rust	879_204	203_040	306_272	71_984	1_625_825

Environment

• dfx 0.23.0
• Motoko compiler 0.13.0 (source 30y2saap-vdivk61d-vl913kh2-srcdk17g)
• rustc 1.76.0 (07dca489a 2024-02-04)
• ic-repl 0.7.5
• ic-wasm 0.7.0

Heartbeat / Timer

Measure the cost of empty heartbeat and timer job.

• setTimer measures both the setTimer(0) method and the execution of empty job.
• It is not easy to reliably capture the above events in one flamegraph, as the implementation detail
  of the replica can affect how we measure this. Typically, a correct flamegraph contains both setTimer and canister_global_timer function. If it's not there, we may need to adjust the script.

Heartbeat

	binary_size	heartbeat
Motoko	142_342	27_506
Rust	23_850	1_112

Timer

	binary_size	setTimer	cancelTimer
Motoko	150_155	56_189	4_702
Rust	504_099	63_379	11_676

Environment

• dfx 0.23.0
• Motoko compiler 0.13.0 (source 30y2saap-vdivk61d-vl913kh2-srcdk17g)
• rustc 1.76.0 (07dca489a 2024-02-04)
• ic-repl 0.7.5
• ic-wasm 0.7.0

Motoko Specific Benchmarks

Measure various features only available in Motoko.

• Garbage Collection. Measure Motoko garbage collection cost using the Triemap benchmark. The max mem column reports rts_max_heap_size after generate call. The cycle cost numbers reported here are garbage collection cost only. Some flamegraphs are truncated due to the 2M log size limit. The dfx/ic-wasm optimizer is disabled for the garbage collection test cases due to how the optimizer affects function names, making profiling trickier.

  • default. Compile with the default GC option. With the current GC scheduler, generate will trigger the copying GC. The rest of the methods will not trigger GC.
  • copying. Compile with --force-gc --copying-gc.
  • compacting. Compile with --force-gc --compacting-gc.
  • generational. Compile with --force-gc --generational-gc.
  • incremental. Compile with --force-gc --incremental-gc.

• Actor class. Measure the cost of spawning actor class, using the Actor classes example.

Garbage Collection

	generate 700k	max mem	batch_get 50	batch_put 50	batch_remove 50
default	1_074_136_336	47_793_792	119	119	119
copying	1_074_136_218	47_793_792	1_073_873_789	1_073_954_095	1_073_875_311
compacting	1_554_238_605	47_793_792	1_200_791_965	1_424_078_246	1_447_969_756
generational	2_326_734_591	47_802_256	899_105_682	1_214_812	1_107_099
incremental	29_505_471	976_097_724	469_026_873	496_491_319	1_282_778_770

Actor class

	binary size	put new bucket	put existing bucket	get
Map	421_218	757_731	16_360	16_930

Environment

• dfx 0.23.0
• Motoko compiler 0.13.0 (source 30y2saap-vdivk61d-vl913kh2-srcdk17g)
• rustc 1.76.0 (07dca489a 2024-02-04)
• ic-repl 0.7.5
• ic-wasm 0.7.0

Publisher & Subscriber

Measure the cost of inter-canister calls from the Publisher & Subscriber example.

	pub_binary_size	sub_binary_size	subscribe_caller	subscribe_callee	publish_caller	publish_callee
Motoko	165_900	150_198	32_864	12_214	27_075	6_629
Rust	538_668	577_244	57_488	37_836	71_326	42_449

Environment

• dfx 0.23.0
• Motoko compiler 0.13.0 (source 30y2saap-vdivk61d-vl913kh2-srcdk17g)
• rustc 1.76.0 (07dca489a 2024-02-04)
• ic-repl 0.7.5
• ic-wasm 0.7.0