End-to-end optimized image compression [PDF]

End-to-end optimized image compression Johannes Ballé Center for Neural Science, NYU Howard Hughes Medical Institute (now with Google Inc.) joint work with: Valero Laparra, Universitat de València Eero P. Simoncelli, CNS/Courant Institute/HHMI International Conference on Learning Representations, Toulon, 2017

2

Linear transform coding �

G

�ˆ

signal space N

G −1

� � ˆ

code space

�∈

M

M

3

Linear transform coding �

G

�ˆ

signal space N

�

G −1

� � ˆ

code space

�∈

M

M

3

Linear transform coding �

G

�ˆ

signal space

G −1

N

�

� � ˆ

code space

�∈

M

M

G

3

Linear transform coding �

G

�ˆ

signal space

G −1

N

�

� � ˆ

code space

�∈

M

M

G

3

Linear transform coding �

G

�ˆ

signal space

G −1

N

�

� � ˆ

code space

�∈

M

M

G

3

Linear transform coding �

G

�ˆ

signal space

G −1

N

�

� � ˆ

code space

�∈

M

M

G

�ˆ

3

Linear transform coding �

G

�ˆ

signal space

D

N

D: R:

G −1

� � ˆ

code space

�∈

M

M

R

distortion, e.g. mean squared error rate, ideally close to Shannon entropy of � 4

rate: 0.17 bits/pixel

5

rate: 0.12 bits/pixel

coarser quantization: lower rate, higher distortion 5

rate: 0.32 bits/pixel

finer quantization: higher rate, lower distortion 5

Linear transform coding � D

G

�ˆ

signal space N

G −1

� � ˆ

code space

�∈

M

M

R

decades of engineering: improved transforms, non-uniform quantization, inter/intra prediction, deblocking, adaptive partitioning, etc.

6

Nonlinear transform coding � D �� , �� :

�ˆ

signal space

�� ��

� � ˆ

code space

� R

multivariate, parametric nonlinear functions (if it helps, think of them as neural networks)

7

Architecture of transformation

convolution

downsampling GDN

downsampling GDN

convolution

downsampling GDN

generalized divisive normalization ��

convolution

�� = � β� +

�� �

2 γ |� | � � ��

8

feature 2

Generalized divisive normalization (GDN)

pointwise regime feature 1 generalization of: – sigmoid-type nonlinearities – local response normalization (LRN) see our ICLR 2016 paper for details 9

feature 2

Generalized divisive normalization (GDN)

pointwise regime feature 1 generalization of: – sigmoid-type nonlinearities – local response normalization (LRN) see our ICLR 2016 paper for details 9

feature 2

Generalized divisive normalization (GDN)

pointwise regime feature 1 generalization of: – sigmoid-type nonlinearities – local response normalization (LRN) see our ICLR 2016 paper for details 9

feature 2

Generalized divisive normalization (GDN)

pointwise regime feature 1 generalization of: – sigmoid-type nonlinearities – local response normalization (LRN) see our ICLR 2016 paper for details 9

feature 2

Generalized divisive normalization (GDN)

joint regime feature 1 generalization of: – sigmoid-type nonlinearities – local response normalization (LRN) see our ICLR 2016 paper for details 10

feature 2

Generalized divisive normalization (GDN)

joint regime feature 1 generalization of: – sigmoid-type nonlinearities – local response normalization (LRN) see our ICLR 2016 paper for details 10

feature 2

Generalized divisive normalization (GDN)

joint regime feature 1 generalization of: – sigmoid-type nonlinearities – local response normalization (LRN) see our ICLR 2016 paper for details 10

Nonlinear transform coding � D

�ˆ

signal space

�

��

�

� ˆ

��

code space

R

optimize �� , �� for rate and distortion numerically L[�� � �� � P� ] = − [log2 P� ] +λ [�(�� �ˆ)] � �� � � �� � R

D

11

gradient is zero almost everywhere

∂ˆ � =0 ∂� ∂ˆ � =∞ ∂�

12

� �

ˆ � +

∆� ∼ �

˜ �

differentiable and continuous stochastic approximation other approaches: Theis et al., 2017 Jang et al., 2017 Maddison et al., 2017 13

� D

L=

��

�ˆ

��

signal space

�

−

� �

� � ˆ

code space

�

� �2 log2 P�� (�� ) + λ ��ˆ − � �2

R

�

14

� D

L=

��

�˜

��

signal space

�

−

� �

proxy loss:

� � ˜

code space

+

� �2 log2 ��˜ � (˜ �� ) + λ ��˜ − � �2

∆�

R

�

14

Wait! Isn’t this just an autoencoder? (Yes and no.)

15

Proxy RD:

L=

Variational AE:

�

rate

−

� �

distortion

� �2 log2 ��˜ � (˜ �� ) + λ ��˜ − � �2

log prior

�

log likelihood

16

D

compression model λ = λ1 R

+

λ1

D

compression model λ = λ2 =

co n

R

st

generative models λ→∞

R +λ 2D = c onst

17

Results

18

original

19

JPEG @ 0.119 bits/px

20

JPEG 2000 @ 0.107 bits/px

21

proposed @ 0.106 bits/px

22

original

JPEG

proposed

JPEG 2000 23

We consistently outperform JPEG 2000

better

better

24

original

25

JPEG @ 0.170 bits/px

26

JPEG 2000 @ 0.167 bits/px

27

proposed @ 0.167 bits/px

28

original

JPEG

proposed

JPEG 2000 29

Thanks!

More images, metrics, and the model parameters: http://www.cns.nyu.edu/~lcv/iclr2017/ Comparison to compression state-of-the-art (BPG): come to our poster tomorrow morning!

30