\(\left\{\begin{matrix}5x^2y-4xy^2+3y^3-2\left(x+y\right)=0\left(1\right)\\xy\left(x^2+y^2\right)+2=\left(x+y\right)^2\left(2\right)\end{matrix}\right.\)

Ta có:

\( (2)\Leftrightarrow xy\left ( x^2+y^2 \right )=x^2+2xy+y^2 \\\ \Leftrightarrow\left (xy-1 \right )\left ( x^2+y^2-2 \right )=0\)

*)TH1: \(xy=1\) thay vào \((1)\) ta được:

\(5x-4y+3y^3-2(x+y)=0\)

\(\Leftrightarrow y^4-2y^2+1=0\)\(\Leftrightarrow y=\pm 1\Rightarrow x=\pm 1\)

*)TH2: \(x^2+y^2=2\).Thay vào \((1)\) ta được:

\(5x^2y-4xy^2+3y^3-(x^2+y^2)(x+y)=0\)

\(\Leftrightarrow 2y^3+4x^2y-5xy^2-x^3=0\)

\(\Leftrightarrow (y^3-x^3)+(y^3+4x^2y-5xy^2)=0\)

\(\Leftrightarrow (y-x)^2(2y-x)=0\)

Với \(x=y\) ta tìm được 2 nghiệm \((x;y)=(1;1); (-1;-1)\)

Với \(x=2y\) thay vào \(x^2+y^2=2\) ta tìm được \(y=\pm \sqrt{\frac{2}{5}}\Rightarrow x=\pm2\sqrt{\frac{2}{5}}\)

Vậy nghiệm của hệ phương trình đã cho là :\((x;y)=(1;1); (-1;-1); \left(2\sqrt{\frac{2}{5}};\sqrt{\frac{2}{5}}\right); \left(-2\sqrt{\frac{2}{5}};-\sqrt{\frac{2}{5}}\right) \)

\(\Leftrightarrow\left\{{}\begin{matrix}x^2\left(xy+1\right)-y\left(xy+1\right)+xy+1=2\\\left(x^2-y\right)^2+xy+1=2\end{matrix}\right.\)

\(\Leftrightarrow\left\{{}\begin{matrix}\left(x^2-y+1\right)\left(xy+1\right)=2\\\left(x^2-y\right)^2+xy+1=2\end{matrix}\right.\)

\(\Rightarrow\left(x^2-y+1\right)\left(xy+1\right)-\left(x^2-y\right)^2-\left(xy+1\right)=0\)

\(\Leftrightarrow\left(xy+1\right)\left(x^2-y\right)-\left(x^2-y\right)^2=0\)

\(\Leftrightarrow\left(x^2-y\right)\left(xy+1-x^2+y\right)=0\)

\(\Rightarrow\left[{}\begin{matrix}y=x^2\\xy+1=x^2-y\end{matrix}\right.\) thay xuống pt dưới:

- Với \(y=x^2\) thay xuống pt dưới \(\Rightarrow x^3=1\)

- Với \(xy+1=x^2-y\) thay xuống dưới:

\(\left\{{}\begin{matrix}xy+1=x^2-y\\2\left(xy+1\right)=2\end{matrix}\right.\) \(\Rightarrow\left\{{}\begin{matrix}xy+1=x^2-y\\xy=0\end{matrix}\right.\)

\(\Rightarrow\left[{}\begin{matrix}x=0;y=-1\\y=0;x^2=1\end{matrix}\right.\)

\(\left\{{}\begin{matrix}xy+x^2=1+y\\xy+y^2=1+x\end{matrix}\right.\Leftrightarrow\left\{{}\begin{matrix}x^2-y^2=y-x\\xy+x^2=1+y\end{matrix}\right.\) ( lấy trên trừ dưới )

\(\Leftrightarrow\left\{{}\begin{matrix}\left(x-y\right)\left(x+y\right)+\left(x-y\right)=0\\xy+x^2=1+y\end{matrix}\right.\)

\(\Leftrightarrow\left\{{}\begin{matrix}\left(x-y\right)\left(x+y+1\right)=0\\xy+x^2=1+y\end{matrix}\right.\)

\(\Leftrightarrow\left\{{}\begin{matrix}\left[{}\begin{matrix}x=y\\x+y=-1\end{matrix}\right.\\xy+x^2=1+y\end{matrix}\right.\) \(\Leftrightarrow\left[{}\begin{matrix}\left\{{}\begin{matrix}x=y\\xy+x^2=1+y\end{matrix}\right.\\\left\{{}\begin{matrix}x+y=-1\\xy+x^2=1+y\end{matrix}\right.\end{matrix}\right.\)

\(\Leftrightarrow\left[{}\begin{matrix}\left\{{}\begin{matrix}x=y\\x^2+x^2=1+x\end{matrix}\right.\\\left\{{}\begin{matrix}x+y=-1\\x\left(x+y\right)-y-1=0\end{matrix}\right.\end{matrix}\right.\) \(\Leftrightarrow\left[{}\begin{matrix}\left\{{}\begin{matrix}x=y\\2x^2-x-1=0\end{matrix}\right.\\\left\{{}\begin{matrix}x+y=-1\\-x-y-1=0\end{matrix}\right.\end{matrix}\right.\)

ta có \(\left\{{}\begin{matrix}x+y=-1\\-x-y-1=0\end{matrix}\right.\left(đúng\right)\)

\(\Leftrightarrow\left\{{}\begin{matrix}x=y\\\left[{}\begin{matrix}x=1\\x=-\dfrac{1}{2}\end{matrix}\right.\end{matrix}\right.\) \(\Leftrightarrow\left[{}\begin{matrix}\left\{{}\begin{matrix}x=1\\y=1\end{matrix}\right.\\\left\{{}\begin{matrix}x=-\dfrac{1}{2}\\y=-\dfrac{1}{2}\end{matrix}\right.\end{matrix}\right.\)

vậy